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CHAPTER  I 


THE  RESEARCH  PROBLEM 

Introduction 

In  recent  years,  reliability  has  been  discussed 
primarily  with  other  logistics  terms  such  as  maintainability 
and  availability.  This  important  triad  of  concepts  which, 
when  linked  together,  form  the  basis  for  operational  readi¬ 
ness  and  combat  effectiveness.  In  speaking  about  the  mission 
of  the  Air  Force  Logistics  Command  (AFLC) ,  its  Commander, 
General  James  P.  Mullins,  says,  "...  our  task  is  to  provide 
quality  products  and  services  on  time  to  the  operational 
commands  [19:29]."  Over  the  last  several  years,  quality  and 
reliability  have  been  used  interchangeably  to  mean  the  same 
thing.  The  Air  Force  defines  quality  as  ",  .  .  the  composite 
of  material  attributes  including  performance  features  and 
characteristics  of  a  product  or  service  to  satisfy  a  given 
need  [26:563]."  An  important  material  attribute  is  relia¬ 
bility  which  the  Air  Force  defines  as  ".  .  .  the  probability 
that  an  item  will  perform  its  intended  function  for  a  speci¬ 
fied  interval  under  stated  conditions  [26:576]."  General 
Mullins,  again,  on  AFLC  says,  "Our  ultimate  objective,  of 
course,  is  to  influence  the  design  of  hardware,  to  design  in 
reliability  and  maintainability  wherever  possible  [19:28]." 


We  can  no  longer  ignore  the  realities  of  technological 
change  and  limited  defense  budgets.  Due  to  the  technological 


expansion  in  the  defense  industry  since  World  War  II, 
Department  of  Defense  (DOD)  weapon  systems  are  more  sophis¬ 
ticated,  complex,  and  expensive.  The  challenge  to  achieve 
increased  reliability  in  these  more  sophisticated  and  complex 
weapon  systems  at  least  cost  requires  innovative  and  creative 
management.  The  DOD  has  accepted  that  challenge  by  implement¬ 
ing  a  contracting  technique  called  Reliability  Improvement 
Warranties  (RIWs) . 


Background 

The  concept  of  RIW  was  presented  to  each  military 
department's  Assistant  Secretary  for  Research  and  Development, 
Installation  and  Logistics  (R&D/I&L)  in  August  1974  in  a 
memorandum  released  by  the  Assistant  Secretary  of  Defense  for 
Installation  and  Logistics  (I&L)  (3:1).  The  memorandum  stated 

that  as  part  of  the  DOD's  efforts  to  improve  operational 
reliability  and  reduce  costs,  RIWs  should  be  used  on  a  trial 
basis  in  the  acquisition  process  in  order  to  help  determine 
the  scope  and  benefits  that  RIWs  might  have  for  the  DOD  and, 
ultimately,  the  Air  Force. 

Previously  known  as  a  "Failure  Free"  or  "Standard 
Warranty,"  the  primary  objective  of  a  RIW  is  to  provide  the 
contractor  with  an  incentive  to  design  and  produce  equipment 


with  high  reliability,  and  to  reduce  repair  costs  during  the 
warranty  period  in  order  to  maximize  contractor  profits 
(29:1-2) . 

Under  the  RIW  concept,  a  contractor  is  obligated, 
within  the  terms  of  the  contract,  to  accomplish  repair  and 
replacement  of  failed  equipment  at  a  firm-fixed  price  for 
the  duration  of  the  warranty  period.  If  equipment  failure 
occurs  during  the  warranty  period,  the  failed  equipment  is 
returned  to  the  contractor  for  repair  and  replacement  at  no 
cost  to  the  user.  Figure  1-1  depicts  how  a  typical  RIW 
contract  functions . 

In  response  to  the  Office  of  the  Secretary  of  Defense 
(OSD)  request  and  based  on  the  basic  RIW  concept,  the  Air 
Force  began  applying  RIWs  to  selected  avionics  systems. 
Through  1982,  Air  Force  RIWs  have  been  applied  on  the  follow¬ 
ing  avionics  subsystems: 

-  AN/ARN  118  TACAN 

-  Carousel  IV  Inertial  Navigation  System  (INS) 

-  C-141  Attitude  Heading  Reference  System  (AHRS) 

-  OMEGA  Navigation  Set 

-  F-16  Electronic  Components  [Radar,  Navigation  Unit, 
Flight  Control  Computer,  Heads  Up  Display  Unit  (HUD) ,  and 
associated  line  replaceable  units  (LRU)] 

-  Air  Force  Standard  Inertial  Navigation  Unit  (INU) 


Fig.  1-1.  RIW  Pipeline 


4 


Since  the  implementation  of  the  Air  Force  RIW  trial 
period ,  little  ongoing  analysis  of  these  subsystems  were  per¬ 
formed  to  determine  the  effectiveness  of  RIWs,  which  was 
indicated  in  a  1979  Air  Force  Audit  Agency  (AFAA)  audit 
report  (1:1-10). 

Problem  Statement 

Since  the  Air  Force ' s  RIW  trial  period  must  be 
closed,  there  was  a  need  to  evaluate  the  effectiveness  of 
RIWs  applied  during  the  trial  period,  and  to  recommend  the 
future  use  of  the  Air  Force  RIW  contracting  concept. 

Justification 

The  Air  Force  RIW  trial  period  began  in  1974  (27:1). 
Periodic  assessments  were  required  during  the  period  to 
determine  if  the  RIW  approach  was  achieving  its  reliability 
and  cost  objectives  (27:20-21).  In  March  1979,  an  AFAA  audit 
report  on  the  Impact  of  New  Management  Concepts  on  Support  and 
Maintainability  of  Avionic  Equipment  identified  weaknesses  in 
the  RIW  approach.  One  relative  weakness  was: 

A  master  plan  for  evaluating  the  impact  of  the 
RIW  concept  on  Air  Force  programs  and  concluding 
the  trial  program  was  not  prepared.  Data  accumu¬ 
lated  for  possible  use  in  the  evaluation  were 
incomplete  and  untimely.  Therefore,  the  ability 
to  assess  the  value  of  existing  RIW  programs  was 
reduced  (1:20]. 

Subsequent  to  this  initial  report,  the  Air  Force  Director  of 
Contracting  and  Manufacturing  Policy  (HQ  USAF/RDC)  was 
established  as  the  Office  of  Primary  Responsibility  (OPR)  for 


r.'J 

r.'- 
.  * 

f.%* 


£ 
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RIW  (1:6).  In  November  1982,  an  AFAA  draft  report  on  F-16 
Weapon  System  Acquisition  and  Logistic  Support  Management 
recommended  that  the  Air  Force  trial  RIW  program  be  concluded, 
and  the  cost  effectiveness  of  meeting  reliability  and  main¬ 
tainability  objectives  be  assessed  (7:2).  In  response  to  this 
second  AFAA  report,  HQ  USAF/RDC  asked  the  Deputy  Chiefs  of 
Staff  for  Contracting  and  Manufacturing  at  both  Headquarters 
Air  Force  Logistics  Command  (HQ  AFLC)  and  Headquarters  Air 
Force  Systems  Command  (HQ  AFSC)  to  provide  recommendations  on 
the  overall  effectiveness  of  the  RIW  approach  and  bring  the 
trial  period  to  a  close  (7:1).  Currently,  within  AFLC  and 
AFSC  respectively,  the  Air  Force  Acquisition  Logistics 
Division  (AFALD)  and  Aeronautical  Systems  Division  (ASD)  are 
evaluating  the  RTW  concept  and  will  provide  their  recommenda¬ 
tions  to  the  Air  Staff  by  31  July  1983.  This  research  report 
will  be  an  independent  evaluation  of  Air  Force  RIW  programs 
during  the  trial  period  and  will  be  an  important  aid  to  Air 
Force  decision  makers  in  evaluating  the  effectiveness  of  the 
RIW  concept. 


Research  Objectives 

1.  To  determine  if  RIWs  applied  to  Air  Force  programs 
since  August  1974  were  effective. 

2.  To  recommend  the  future  use  by  the  Air  Force  of 
the  RIW  concept. 
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1.  What  are  the  perceptions  of  active  duty  Air 
Force  officers  and  DOD  civilians  with  Air  Force  RIW  expertise 
regarding  the  effectiveness  of  RIWs  as  applied  to  Air  Force 
programs  since  August  1974? 

2.  What  is  the  recommendation  for  the  future  use  by 
the  Air  Force  of  the  RIW  concept,  as  perceived  by  Air  Force 
officers  and  DOD  civilians  with  Air  Force  RIW  expertise? 

Supporting  Qualitative  Research 
Hypotheses 

The  research  hypotheses  to  answer  Research  Question 

One  ares 

1-6.  Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  perceive  the 

(1)  AN/ARN-118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  AF  Standard  INU 
RIW  program  as  ineffective. 

7.  Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  perceive  the  Air  Force  RIW 
program  as  ineffective. 

The  research  hypothesis  to  answer  Research  Question 


Two  is: 


8.  Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  do  not  recommend  the  continued 


use  by  the  Air  Force  of  the  RIW  concept. 

General  Research  Plan 

This  section  presents  a  brief  overview  of  the  general¬ 
ized  approach  to  the  research.  The  general  research  plan  con¬ 
centrates  on  the  perceptions  of  active  duty  Air  Force  officers 
and  DOD  civilians  with  Air  Force  RIW  expertise  regarding 
effectiveness  of  the  Air  Force  RIW  program.  The  instrument 
for  obtaining  these  perceptions  is  a  formal  interview.  The 
interview  is  divided  into  the  following  four  sections:  General/ 
Demographic,  Program  Evaluation,  Future  Recommendation,  and 
Open-Ended  Questions.  Responses  to  the  Program  Evaluation  sec¬ 
tion  are  assigned  to  the  research  hypothesis  associated  with 
Research  Question  One;  while  responses  to  the  Future  Recommenda¬ 
tion  section  are  assigned  to  the  research  hypothesis  associated 
with  Research  Question  Two.  To  evaluate  each  of  these  research 
hypotheses,  nonparametric  statistical  tests  are  used  to  measure 
the  perceptions  of  the  respondents.  Finally,  the  General/ 
Demographic  section  is  used  to  document  the  credibility  of  the 
respondents,  and  the  Open-Ended  Questions  section  is  used  to 
support  the  two  research  questions. 

Through  the  use  of  this  general  research  plan,  this 
research  team  will  develop  supporting  hypotheses  to  answer  the 
two  research  questions.  Answers  to  these  questions  will  meet 
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the  stated  research  objectives  which,  in  turn,  will  address 
the  problem  statement  (Figure  1-2) . 


Definitions 


The  following  terms  used  throughout  this  report  are 
defined  as  follows: 


Life  Cycle  Cost  (LCC) 

The  LCC  of  a  system  is  the 

.  .  .  sum  total  of  the  direct,  indirect,  re¬ 
curring,  and  nonrecurring,  and  other  related  costs 
incurred,  or  estimated  to  be  incurred,  in  the  design, 
development,  production,  operation,  maintenance  and 
support  of  (that)  system  over  its  anticipated  use¬ 
ful  life  span  [29:3]. 


Mean  Time  Between  Failure  (MTBF) 

A  time  expressed  in  hours  quantified  by  dividing 
mean  operating  hours  by  the  mean  number  of  failures. 

Reliability  Improvement  Warranty  (RIW) 

A  provision  in  either  a  fixed  price  acquisition,  or 
fixed  price  equipment  overall  contract  in  which: 

1.  The  contractor  is  provided  with  a  monetary  incen¬ 
tive  (by  being  allowed  to  retain  all  savings  accrued  due  to 
reduced  contractor  support  costs  during  the  period  of  warranty) 
throughout  the  period  of  the  warranty  to  improve  the  production 
design,  and  engineering  of  the  equipment  so  as  to  enhance  the 
field/operational  reliability  and  maintainability  of  the 
system/equipment . 


2.  The  contractor  agrees  that,  during  a  specific  or 
measured  period  of  use,  he  will  repair  or  replace  (within  a 


specified  turnaround  time)  all  equipment  that  fails  (subject 
to  specified  exclusions  if  applicable)  (27:5-6). 

Turnaround  Time 

Time  interval  from  receipt  of  a  failed  unit  at  the 
manufacturer's  dock  until  receipt  of  the  repaired  unit  at  a 
Government-bonded  warehouse. 

Warranty 

A  warranty  is  a  promise  of  affirmation  given  by  a 
seller  to  a  purchaser  regarding  the  nature,  usefulness,  or 
condition  of  the  supplies  or  performance  of  services  to  be 
furnished.  The  principal  purposes  of  a  warranty  in  a 
Government  contract  are  to  delineate  the  rights  and  obliga¬ 
tions  of  the  contractor  and  the  Government  for  defective  items 
and  services  and  to  foster  quality  performance.  Generally, 
warranties  survive  acceptance  of  the  contract  items  for  a 
stated  period  of  time  or  use,  or  until  the  occurrence  of  a 
specified  event,  notwithstanding  other  contractual  provisions 
pertaining  to  acceptance  by  the  Government  (27 : Sec. 2 : 55) . 

RIW  with  MTBF  Guarantee 

A  basic  RIW  contract  with  the  added  incentive  to 
assure  that  the  contractor's  equipment  achieves  a  specified 
MTBF  by  the  end  of  the  RIW  period.  The  MTBF  guarantee  may 
include  provisions  for  reliability  growth  (increase  MTBF) 
as  the  equipment  matures. 
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Effectiveness 


The  quality  or  state  of  being  able  to  produce  a 
definite  or  desired  result  as  indicated  by  increased  relia¬ 
bility,  maintainability,  contractor  competition,  use  rate, 
and  RIW  contract  clause,  and  reduced  acquisition  cost, 
equipment  modification,  data  collection,  supply  system, 
maintenance  skills,  and  number  of  spares. 

Research  Report  Overview 

As  an  overview  of  this  research  report,  the  content  of 
the  remaining  chapters  is  briefly  discussed. 

Chapter  II  is  a  review  of  the  literature  applicable 
to  the  subject  of  reliability  improvement  warranties.  This 
literature  describes  RIWs  in  the  Commercial  Airline  industry, 
in  the  Department  of  Defense,  and  in  the  United  States  Air 
Force.  Literature  discussing  past  and  present  Air  Force  RIW 
programs  is  reviewed.  Finally,  this  chapter  reviews  the 
research  which  discusses  the  effectiveness  of  RIWs. 

Chapter  III  describes  the  methodology  used  to  accomplish 
the  research  objectives  and  answer  the  research  questions 
discussed  in  Chapter  I.  This  chapter  describes  the  population 
from  which  data  were  gathered,  the  interview  instrument  used 
to  collect  the  data,  and  the  procedures  to  process  and  analyze 
the  data.  Finally,  scope,  limitations,  and  assumptions  of  the 
research  are  described. 
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Chapter  IV  contains  an  analysis  of  the  data  collected 
by  the  survey  instrument.  The  procedures  described  in 
Chapter  III  will  be  used  to  process  and  analyze  each  iteration 
of  the  survey  instrument. 

Chapter  V  summarizes  the  conclusions  that  can  be 
drawn  from  this  study  of  the  effectiveness  of  Air  Force  RIWs 
during  the  trial  period.  Recommendation  for  future  use  of 
Air  Force  RIWs  are  made,  and  recommendations  for  future 


research  are  suggested. 


CHAPTER  II 


LITERATURE  REVIEW 

Introduction 

The  Department  of  Defense  (DOD)  has  focused  attention 
on  assuring  that  our  defenses  are  prepared  to  fight  with  the 
most  technically  up-to-date  and  reliable  equipment.  With  the 
sophistication  and  complexity  of  DOD  weapon  systems  increasing, 
the  challenge  to  insure  readiness  and  system  availability  is 
through  increasing  the  reliability  of  our  systems  at  the  least 
possible  cost.  The  focus  of  much  of  this  challenge  is  on 
those  systems  and  subsystems  which  most  often  lead  to  poor 
availability  because  of  deficiencies  in  reliability  and  main¬ 
tainability  (20:1).  This  emphasis  cannot  be  understated  nor 
taken  for  granted.  Senator  William  Proxmire  (D-Wis) ,  in 
prefacing  the  testimony  of  a  recent  Central  Intelligence 
Agency  report,  says 

The  Soviet  Union  has  built  an  exceedingly  power¬ 
ful  military  force.  Under  Kruschev  the  emphasis  was 
on  strategic  nuclear  programs,  but  Brezhnev  presided 
over  an  across-the-board  expansion  and  modernization 
of  all  Soviet  forces  [25:4]. 

One  effort  to  curb  the  rising  costs  of  weapon  systems 
and,  at  the  same  time,  insure  operationally  ready  equipments 
to  meet  the  Soviet  threat,  the  DOD  has  implemented  a  concept 
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known  as  the  Reliability  Improvement  Warranty.  This  concept 
is  an  outgrowth  of  historical  experience  with  the  commercial 
airlines . 

Commercial  Airline  Experience 
The  airlines  use  a  form,  fit  and  function  interchange- 
ability  concept  for  much  of  their  avionics  equipment.  Since 
the  1930s,  this  has  been  part  of  the  commercial  airline 
acquisition  methodology  (CAAM) ,  a  primary  feature  is  to  use 
warranty  clauses  in  acquisition  contracts  (16:16).  Pan 
American  Airlines,  Inc. ,  with  over  $1  billion  invested  in 
747  aircraft,  wanted  to  reduce  the  high  support  cost  risks 
associated  with  poor  equipment  reliability  in  their  aircraft 
fleet  (16:11).  In  doing  so.  Pan  American  specified  a 
guaranteed  mean  time  between  failure  (MTBF)  which  would  be 
achieved  by  the  end  of  the  five-year  warranty  period  (16:12). 
As  a  result  of  this  airline  experience.  Aeronautical  Radio, 
Incorporated  (ARINC) ,  the  communications  advisory  board  of 
the  U.S.  Air  Transport  Industry,  developed  a  list  of  CAAM 
guidelines  which  form  the  present  basis  of  the  DOD  RIW  guide¬ 
lines.  While  the  warranty  approach  was,  in  theory,  a  sound 
one,  the  Rand  Corporation  reported  in  1977  that  the  airlines 
were  unable  to  reduce  life-cycle  costs  by  use  of  a  warranty 
(10:43).  The  report  goes  on  to  say,  ".  .  .  the  justification 
of  warranty  usage  in  the  military  on  the  basis  of  better 
MTBFs  in  commercial  avionics  is  misleading  [10:43].” 


Notwithstanding  this  report,  the  DOD  implemented  the  use  of 
warranties  in  weapon  system  acquisitions  to  improve  relia¬ 
bility. 


Failure  Free  Warranty  (FFW) 

The  emphasis  on  reliability  evolved  out  of  the  early 
1960s,  in  which  a  joint  Air  Force  Logistics  and  Systems 
Command  concept  to  determine  the  true  cost  of  procurement, 
was  expanded  to  include  a  product- life  warranty  system  in 
the  life-cycle  cost  area  (21:7).  This  led  to  a  proposal  from 
the  Instruments  Division  of  Lear  Siegler,  Incorporated  (LSI) 
to  repair  or  replace  MD-1  vertical  gyros  under  a  failure  free 
warranty  concept.  The  proposal  was  not  accepted  by  the  Air 
Force,  but  it  did  set  the  foundation  for  use  by  the  Navy. 

In  1967,  the  Navy  awarded  the  first  FFW  contract  to 
Lear-Siegler  to  repair  gyroscope  platforms  on  Navy  A-4  and 
F-4  aircraft  (22:12).  The  LSI  proposal  included  a  fixed 
price  for  repairing  gyros  over  a  specified  operating  time 
period.  It  was  subsequently  reported  that  over  the  five-year 
warranty  period  the  MTBF  increased  33  percent  and  repair 
costs  were  reduced  by  40  percent  (17:51).  By  1969,  the  Air 
Force  implemented  a  FFW  concept  on  the  F-lll  displacement 
gyro.  Terms  of  the  contract  specified  3,000  operating  hours 
or  five  years,  whichever  came  first  (22:12A) .  Under  this 
contract,  acquisition  costs  were  reduced  37  percent.  The 
warranty  price  over  the  five-year  period  was  7  percent  of  the 


per  unit  acquisition  cost  (22:18).  Reliability  of  the 
gyros,  however,  decreased  over  the  warranty  period  for  unknown 
reasons  (22:19-20).  Realizing  the  potential  for  reliability 
growth  and  reduced  support  costs,  the  DOD  implemented  the  RIW 
trial  period  in  1973. 

Department  of  Defense  Trial  Use  of  RIW 
The  DOD  trial  period  was  initiated  in  August  1973  with 
an  Assistant  Secretary  of  Defense  for  Installations  and 
Logistics  (I&L)  memorandum  which  directed  trial  use  of  RIW  by 
the  services  ".  .  .in  the  acquisition  and  initial  operational 
support  of  a  number  of  Electronic  Subsystems  [4:1]."  Follow¬ 
ing  the  ASD  (I&L)  memorandum,  a  Reliability  Improvement 
Warranty  Committee  was  formed  to  prepare  definitive  OSD 
policy  on  RIW  applications  (3:2).  In  essence,  this  committee 
was  responsible  for  formulating  the  RIW  guidelines.  In 
August  1974,  the  Assistant  Secretary  of  Defense  for  Installa¬ 
tions  and  Logistics  sent  the  guidelines  to  all  three  services 
for  implementation  (3:1-20).  An  important  concept  in  the  RIW 
approach,  as  stated  in  the  memo,  was: 

.  .  .  thus,  the  intent  of  the  RIW  contracting 
technique  is  to  realize  improved  operational 
reliability  and  maintainability  of  DOD  systems 
and  equipments  for  each  additional  dollar  that  the 
contractor  uses.  For  these  reasons,  a  RIW  is  not  a 
maintenance  contract  and,  therefore,  should  not  be 
used  for  this  purpose  [3:1]. 


In  response  to  this  direction  from  DOD,  the  Air  Force 
established  interim  guidelines  to  implement  this  concept. 


Air  Force  Trial  Use  of  RIW 


In  1964,  Air  Force  Operational  and  Maintenance  (O&M) 
costs  accounted  for  21  percent  of  the  Air  Force  budget;  and 
in  1973,  O&M  costs  were  responsible  for  27  percent  of  the 
Air  Force  budget  (ll:p.6-2).  With  Air  Force  O&M  costs  rising 
and  defense  budgets  constrained,  it  became  apparent  that 
acquisition  and  support  costs  must  be  reduced.  In  July  1974, 
the  Directorate  of  Procurement  Policy  at  HQ  USAF  published 
the  "Interim  Reliability  Improvement  Warranty  (RIW)  Guide¬ 
lines."  The  primary  emphasis  behind  the  trial  period,  as 
stated  in  the  guidelines,  is  as  follows: 

The  essence  of  the  RIW  philosophy  is  that 
during  the  period  of  the  warranty  coverage,  for  a 
fixed  price,  contractors  will  be  encouraged  to 
improve  the  reliability  and  to  reduce  the  repair 
costs  of  their  equipment  through  the  mechanism  of 
"no-cost"  (to  the  Government)  Engineering  Change 
Proposals  (ECPs) .  These  ECPs  shall  be  consistent 
with  Government  procedures  to  preserve  Configuration 
Control.  Once  a  fixed  price  is  established  for  the 
RIW,  the  actual  profit  realized  by  the  contractor 
is  dependent  upon  the  equipments'  reliability  and 
maintainability  in  service  use,  plus  any  improve¬ 
ments  that  he  can  make  in  its  reliability  and 
maintainability  so  as  to  keep  the  number  and  cost 
of  repairs  as  low  as  possible  [27:4], 

The  emphasis  of  the  RIW  is  to  focus  contractor  attention  on 

designed-in  reliability  and  maintainability.  The  guidelines, 

thus,  go  on  to  say, 

...  as  RIW  becomes  a  contracting  technique 
by  which  the  Government  derives  the  benefits  of 
improved  reliability  and  maintainability  for  each 
additional  dollar  that  the  contractor  earns  [27:4-5]. 


Prior  to  use  of  the  RIW  approach,  contractors  had  little 
incentive  to  improve  reliability.  As  can  be  seen  in  Figure 
2-1,  the  contractor,  under  a  non-RIW  situation,  will  be 
motivated  to  design  and  produce  at  minimum  specified  relia¬ 
bility  beyond  required  specifications  (14:24).  However, 
with  the  RIW  approach,  the  cost  curves  which  go  into  describ¬ 
ing  overall  profit  for  the  contractor  are  shown  in  Figure 
2-2.  It  should  be  noted  that  now  the  support  costs,  as  well 
as  initial  production  costs,  effect  total  costs  of  the 
contractor.  In  this  case,  the  management  of  costs  associated 
with  the  RIW  are  transferred  to  the  contractor.  From  the 
total  cost  curve  in  Figure  2-2,  the  contractor  can  maximize 
profit  by  making  appropriate  trade-offs  between  cost  and 
reliability.  Assuming  a  constant  contract  price  of  the 
buyer  (the  Government) ,  the  contractor  will  produce  where 
profit  is  maximum — at  the  minimum  point  (A)  on  the  total 
cost  curve  and,  at  this  point,  reliability  (MTBF)  is  greater 
than  the  minimum  acceptable  reliability.  It  can  be  seen 
that  the  contractor  is  encouraged  to  introduce  design  changes 
early  in  production,  which  can  result  in  higher  initial  costs, 
but  can  be  offset  by  reduced  support  costs  (2:15). 

The  guidelines  also  set  forth  a  number  of  criteria 
against  which  potential  systems  and  subsystems  must  be 
evaluated  to  determine  if  they  lend  themselves  to  RIW  applica¬ 
tion.  The  guidelines  were  tailored  for  application  to 
avionics  equipments.  This  is  due,  in  part,  to  the  expense. 


Costs 


Minimum  acceptable 
.  reliability 


Fig.  2-1.  Non-RIW  Profit  Motive  (14:24) 


Costs 


Minimum  acceptable 
,  reliability 

!  Total  cost 


complexity  and  sophistication  of  avionics  equipment,  as  well 
as  the  Air  Force  investment  of  nearly  $12  billion  in  this 
equipment  which  must  be  maintained  and  supported  (1:1).  As 
stated  in  the  RIW  guidelines,  the  applications  criteria  are 
as  follows: 

1.  A  warranty  can  be  obtained  at  a  price  commensurate 
with  the  contemplated  value  of  the  warranty  work  to  be 
accomplished,  with  consideration  being  given  to  the  contrac¬ 
tually  specified  R&M  requirements. 

2.  Moderate  to  high  initial  support  costs  are  involved. 

3.  The  equipment  is  readily  transportable  to  permit 
return  to  the  vendor's  plant  or,  alternatively,  the  equipment 
is  one  for  which  a  contractor  can  provide  field  service. 

4.  The  equipment  is  generally  self-contained,  is 
generally  immune  from  failures  induced  by  outside  units,  and 
has  readily  identifiable  failure  characteristics. 

5.  The  equipment  application  in  terms  of  expected 
operating  time  and  the  use  environment  are  known. 

6.  The  equipment  is  susceptible  to  being  contracted 
for  on  a  fixed  price  basis,  with  competition  on  the  basis  of 
form,  fit  and  function  stimulated  to  the  extent  practicable. 

7.  The  contract  can  be  structured  to  provide  a 
warranty  period  of  from  three-five  years.  This  should  allow 
the  contractor  sufficient  time  to  identify  and  analyze 
failures  in  order  to  permit  reliability  and  maintainability 
improvements . 


8.  The  equipment  has  a  potential  for  both  relia¬ 
bility  growth  and  reduction  in  repair  costs. 

9.  Potential  contractors  indicate  a  cooperative 
attitude  toward  acceptance  of  an  RIW  provision  and  evaluation 
of  its  effectiveness. 

10.  A  sufficient  quantity  of  the  equipment  is  to  be 
procured  in  order  to  make  the  RIW  cost-effective. 

11.  The  equipment  is  of  a  configuration  that  dis¬ 
courages  unauthorized  field  repair,  preferably  sealed  and 
capable  of  containing  an  Elapsed  Time  Indicator  (ETI)  or  some 
other  means  of  usage  indication. 

12.  There  is  a  reasonable  degree  of  assurance  that 
there  will  be  a  high  utilization  of  the  equipment. 

13.  The  equipment  is  one  that  permits  the  contractor 
to  effect  no-cost  ECPs  subsequent  to  the  Government's 
approval. 

14.  Failure  data  and  the  intended  operational  use 
data  can  be  furnished  the  contractor  for  the  proposed  con¬ 
tractual  period  and  updated  periodically  during  the  term  of 
the  contract. 

It  is  noted  that  the  equipment  need  not  meet  all  the 
criteria  shown  above  in  order  to  apply  an  RIW.  Rather,  at 
this  point  in  time,  we  should  pick  logical  candidates  meeting 
several  or  many  of  the  criteria  so  that  further  assessment 
can  be  made  of  the  value  of  this  technique  (27:15-16). 
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It  is  not  general  practice  in  the  Air  Force,  or  DOD 
for  that  matter,  to  contract  separate  systems,  one  with  RIW 
and  one  without,  to  determine  which  approach  is  the  most  cost 
effective.  For  this  reason,  the  RIW  guidelines  state, 

...  a  cost  analysis  should  be  performed  for 
each  proposed  warranty  application,  upon  receipt  of 
the  contractor's  proposal,  in  order  to  determine 
whether  the  use  of  an  RIW  would  be  cost-effective. 

Such  an  analysis  should  investigate  the  relative 
cost  of  the  RIW  and  non-RIW  situations  (including 
ECPs)  and  examine  the  cost  of  varying  time 
periods  [27:12]. 

As  part  of  the  overall  cost  benefit  analysis,  in  addition  to 
knowing  firm  contractor  costs  for  the  RIW,  it  is  necessary  to 
determine  what  (the  Air  Force)  would  have  paid  in  support 
costs  for  the  same  system  under  a  non-RIW  application. 

These  cost  analyses,  thus,  become  the  basis  for  the  compari¬ 
sons  to  determine  the  most  cost-effective  approach  over  the 
total  life-cycle  of  the  system  or  subsystem.  These  guide¬ 
lines  established  the  foundation  for  the  Air  Force  programs 
which  have  applied  the  RIW  concept. 


Air  Force  Experience  with  RIW 
Since  the  trial  period  began  for  the  Air  Force  in 


1974,  six  programs  have  implemented  the  RIW  concept.  Of  the 


six,  the  F-16  System  Program  Office  warranted  nine  Line 
Replaceable  Units  (LRUs)  through  a  single  contract  with  the 
prime  airframe  contractor,  General  Dynamics.  Table  2-1 
shows  the  program  breakout  by  contractor  and  warranty  period. 

In  July  1975,  the  AN/ARN-118  TACAN  was  the  first 
Air  Force  program  to  use  the  RlW-type  contract.  The  contract 
specified  a  minimum  reliability  of  500  hours  MTBF  approximately 
two  years  after  contract  award.  Reliability  growth  to  800 
hours  MTBF  was  expected  within  46-69  months  after  contract 
award  (21:22). 

In  November  1975,  the  Air  Force  awarded  its  second 
major  RIW  contract  for  the  Carousel  IV  inertial  navigation 
system  (INS)  to  Delco  Electronics.  The  minimum  acceptable 
MTBF  for  proposal  purposes  was  750  hours;  Delco  bid  1,128 
hours  (21:28). 

In  1976,  the  F-16  program  initiated  a  major  effort 
on  the  part  of  the  Air  Force  in  awarding  a  RIW  contract  to 
General  Dynamics.  As  noted  in  Table  2-1,  nine  LRUs  were 
covered  by  the  contract.  A  unique  aspect  of  the  program  is 
that  the  RIW  applied  not  only  to  the  first  250  Air  Force 
aircraft  installs,  but  to  the  first  192  installs  in  those 
aircraft  purchased  by  the  European  Participating  Governments 
(13:p.4-l).  Of  the  nine  LRUs,  only  two  contain  a  RIW  with  MTBF 
guarantee — the  Radar  Transmitter  and  the  Heads-up-display 
(HUD)  Electronics. 


TABLE  2-1 


MAJOR  AIR  FORCE  APPLICATIONS 
OF  RIW.  (13:40) 


Program 

Contractor 

Warranty 

Period 

AN/ARN-11S  TACAN 

Collins  Avionics 
Rockwell  Int'l 

5  yr 

C-141  AHRS 

Lear-Siegler ,  Inc. 

5  yr 

OMEGA  Nav  Set 

Dynell  Electronics 

5  yr 

Carousel  IV  INS 

Delco  Electronics 

5  yr 

F-16  Electronics 

General  Dynamics 

4  yr 

Fit  Control 

Computer 

Lear-Siegler 

or 

300,000 

Radar  Antenna 

Westinghouse 

flight  hours 
(whichever 

Radar  lo  Power  RF 

Westinghouse 

comes  first) 

Radar  Digital 
Processor 

Westinghouse 

Radar  Computer 

Westinghouse 

Radar  Transmitter 

Westinghouse 

Heads-Up-Display 

Marconi-Elliot 

HUD  Electronics 

Marconi-Elliot 

Inertial  Nav 

System 

Singer-Kearfott 

AF  Standard  INU 


Litton  Guidance 
and  Control 


5  yr 


In  a  more  recent  trial  of  RIW,  the  Air  Force's 


Standard  Inertial  Navigation  Unit  (INU)  awarded  a  five-year 
RIW  with  MTBF  guarantee  to  Litton  Industries,  Guidance  and 
Control  Division.  Reliability  at  the  first  measurement 
period  (one  year  after  delivery  of  the  first  production  INU) 
was  to  be  275  hours  MTBF.  Growth  over  five  years  is  to 
achieve  reliability  of  525  hours,  after  three  more  measurement 
periods  over  the  warranty  period. 

According  to  a  1979  report  (24:68),  the  RIW  trial 
phase  has  indicated  measures  of  success  with  increases  in 
MTBFs  at  reduced  costs.  The  report  goes  on  to  say,  however, 
that  an  evaluation  of  the  effectiveness  of  the  RIW  is  yet 
to  be  made. 


Effectiveness  of  Riw 

Since  the  Air  Force  followed  suit  with  the  airlines' 
practice  of  using  warranties  for  avionics,  the  researchers 
would  expect  lessons  on  how  and  what  to  measure  to  determine 
RIW  effectiveness.  But,  as  a  1979  Rand  report  points  out, 
warranties  and  their  effects  are  not  completely  understood 
even  by  the  airlines  and,  as  such,  "...  they  have  developed 
no  standard  by  which  to  measure  the  cost-benefit  derived  from 
warranties  [10:2]."  In  the  same  report,  observations  on  the 
DOD  RIW  trial  period  were  made.  The  authors  suggest  that  the 
value  of  RIW  over  the  trial  period  will  yield  inconclusive 
evidence  due  primarily  to  varying  contractual  complexities 


26 


between  programs  and,  more  importantly,  the  lack  of  a  well- 
defined  plan  by  which  the  RIW  can  be  evaluated  (10:54). 

Along  the  same  lines,  a  RIW  workshop,  reported  that  it  was  the 
contractual  issues,  not  technical,  which  posed  the  greatest 
problems  for  the  RIW  (8:2). 

In  a  revealing  analysis  which  evaluated  and  recom¬ 
mended  changes  to  the  RIW  guidelines,  Jacobson  and  Skaggs 
concluded  that  inadequacies  in  the  existing  guidelines  may 
limit  overall  effectiveness  of  the  RIW  approach  (16:64-69). 
According  to  a  1976  report,  industry  believes  that,  while  the 
DOD  is  serious  about  the  use  of  RIW  now  and  in  the  future, 
the  RIW  trial  period  should  be  suspended  until  the  guidelines 
can  be  revised.  Their  concern  centers  around  the  implied 
use  of  RIW  for  any  new  equipment  whose  design  may  use  a  new 
technology  without  the  benefit  of  prior  field  experience  (5:11) 


Conclusions  from  the  Literature 
In  reviewing  the  literature  on  the  Reliability  Improve 
ment  Warranty  concept  and  its  impacts,  many  parallels  were 
observed  among  research  reports.  What  follows  are  this 
research  team’s  three  most  prevalent  conclusions  drawn  from 
this  review. 

1.  Reliability  Improvement  Warranties  are  not  a 
cure-all  for  Air  Force  reliability  and  maintainabili cy 
problems  but,  when  applied,  implemented,  administered,  and 
evaluated  in  accordance  with  the  principles  of  the  RIW 


guidelines,  have  the  potential  for  achieving  reliability, 
maintainability  and  cost  objectives. 

2.  Neither  the  Government  nor  industry  fully  appre¬ 
ciate  the  RIW  concept  to  the  extent  required  for  both  parties 
to  realize  benefits  therefrom. 

3.  There  is  no  "watch  dog"  within  the  Air  Force  to 
monitor  the  progress  of  RIW  throughout  the  trial  period. 

While  theoretically  sound  policy  was  disseminated,  there  has 
been  little  follow-up  action  to  assure  compliance  with  the 
direction  and  policy. 

The  research  which  follows  is  not  intended  to  solve 
the  RIW  effectiveness  issue,  but  merely  shed  some  new  light 
on  a  somewhat  clouded  problem  area. 
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CHAPTER  III 


RESEARCH  METHODOLOGY 

Introduction 

In  Chapter  I,  the  basic  problem  and  formulated 
research  questions  were  defined.  This  chapter  outlines  the 
methodology  used  in  answering  the  research  questions.  The 
research  team  identifies  and  interviews  experts  in  the  field 
of  RIW,  develops  an  interview  instrument  to  gather  the  data, 
and  describes  the  statistical  and  descriptive  analyses  to 
be  performed  on  the  data  gathered  by  this  research.  Finally, 
the  assumptions,  scope  and  limitations  of  this  research 
effort  are  presented. 

Identification  and  Interview 
Procedure 

Identification  of  RIW  Experts 

The  first  step  in  the  process  was  to  identify  those 
individuals  with  Air  Force  RIW  expertise.  Individuals  were 
chosen  from  two  target  populations:  (1)  active  duty  Air  Force 
officers  and  DOD  civilians  currently  involved  in  one  • f  the 
six  previously  identified  Air  Force  RIW  programs;  and  (2) 
active  duty  Air  Force  officers  and  DOD  civilians  previously 
involved  in  one  of  the  six  Air  Force  RIW  programs.  The  sample 


was  drawn  from  these  two  target  populations  and  consisted 
of  individuals 


1.  whose  opinions  and  judgment  were  sought; 

2.  from  the  disciplines  of  program  management, 
contracting  and  logistics  management;  and 

3.  who  were  willing  to  participate  in  the  research. 
From  the  drawn  sample,  two  criteria  were  used  to  identify 
the  experts: 

1.  They  must  have  a  basic  knowledge  of  the  problem 
area  and  be  able  to  apply  that  knowledge. 

2.  They  must  have  experience  formulating,  implement¬ 
ing  or  monitoring  an  Air  Force  RIW  program. 

It  was  assumed  that  two  years  in  program  management,  contract 
management,  or  logistics  management  on  any  of  the  six  identi¬ 
fied  Air  Force  RIW  programs  yielded  the  experience  necessary 
to  consider  the  individuals  knowledgeable  of  Air  Force  RIW 
and,  thereby,  were  qualified  under  both  criteria  above. 

From  the  selected  sample  group,  identified  individuals  who 
met  the  above  criteria  were  considered  qualified  to  participate 
in  this  research. 

Contacting  RIW  Experts 

The  next  step  in  the  process  was  to  contact  those 
individuals  qualified  to  participate  in  this  research. 

Initial  contact  with  qualified  individuals  was  by  telephone 
to  obtain  their  approval  to  participate  in  this  effort. 


After  obtaining  approval  to  participate  in  the 


research  effort,  and  prior  to  a  structured  interview,  the 
following  two  actions  were  accomplished: 

1.  Preparation  of  an  information  package  (Appendix  A) 
which  included  the  purpose  of  the  study,  the  primary  problem/ 
questions  to  be  analyzed,  the  expert's  role  in  reaching  a 
solution  and  the  uniqueness  of  their  abilities  in  the  total 
effort  (18:54) . 

2.  Preparation  of  an  interview  plan  which  included 
the  finalization  and  standardization  of  the  structured  inter¬ 
view  procedures,  and  the  development  of  the  interview 
schedule. 

Once  the  information  packages  were  sent  to  the  par¬ 
ticipating  individuals  (hereafter  identified  as  the  RIW 
experts) ,  the  researchers  contacted  the  RIW  experts  within 
one  week  to  insure  they  received  the  information  and  to 
schedule  the  structured  interviews. 

The  Structured  Interview 

Two  means  of  conducting  the  structured  interview 
were  used.  The  first  means  was  a  personal  interview  with  the 
RIW  experts  located  at  Wright-Patterson  AFB ,  Ohio.  Due  to 
cost  and  time  constraints,  the  second  means  was  through 
telephone  contact  with  the  RIW  experts  geographically  located 
away  from  Wright-Patterson  AFB.  To  insure  minimum  bias 
between  the  personal  and  telephone  interviews,  a  standardized 
interview  format  was  used. 


During  the  structured  interview,  the  researchers 


used  the  following  standardized  format: 

1.  Obtain  demographic  information  from  the  RIW 

expert . 

2.  Ascertain  complete  understanding  by  the  RIW 
expert  of  the  research  problem,  the  research  questions  and 
measurement  scales,  and  the  effectiveness  criteria. 

3.  Ask  the  interview  questions  and  record  the  RIW 
expert's  ratings  of  the  effectiveness  criteria  and  future 
recommendation . 

4 .  Conclude  with  open  comments  which  the  RIW 
expert  feels  may  be  pertinent  to  the  research. 

The  Interview  Instrument 


General 

After  considering  a  number  of  techniques  to  gather 
data,  the  research  team  determined  that  a  structured  inter¬ 
view  would  yield  results  desired  for  this  research  effort. 

In  making  this  determination,  primary  consideration  was  given 
to  the  location  of  RIW  experts,  cost  and  time  to  collect 
data.  There  were  several  advantages  to  using  a  structured 
interview,  including: 

1.  An  interview  structured  around  a  list  of  specific 
criteria  would  result  in  a  more  meaningful  response. 

2.  Depth  and  detail  of  information  which  could  be 

secured. 
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3.  Quality  of  information  could  be  improved. 

4.  Interviewer  had  more  control  over  the  interview. 


5.  Timeliness  and  cost-effectiveness. 

6.  Reduction  of  nonresponse  bias  (9:294,305). 

Based  on  these  advantages  and  primary  considerations,  a 
combination  of  personal  and  telephone  interviews  was  used. 

With  respect  to  this  decision,  Emory  identified  several 
disadvantages  in  the  use  of  the  personal /telephone  interview. 
These  disadvantages  included: 

1.  Limits  on  the  length  of  the  interview. 

|  2.  Adverse  effects  of  interviewers  who  alter  the 

questions  asked. 

3.  Reluctance  to  talk  with  strangers  (9:294,306). 

This  research  effort  insured  anonymity  of  the  RIW 
experts.  The  purpose  of  this  anonymity  was  to  relieve  the 
RIW  experts'  concern  that  candid  and  honest  answers  would 
affect  their  professional  careers  (6:43). 

Development 

In  order  to  obtain  the  desired  information  from  the 
RIW  experts,  tne  interview  instrument  was  developed  in  four 
sections  (Appendix  B) .  The  first  section,  General /Demographic 
was  used  for  the  following  two  purposes: 

1.  encourage  response  and  promote  rapport,  and 

2.  document  the  credibility  of  the  RIW  experts. 


The  second  section,  Program  Evaluation,  measured  the 


effectiveness  of  each  RIW  program  as  perceived  by  the  RIW 
experts.  In  measuring  these  programs'  effectiveness,  the  RIW 
experts  rated  the  eleven  effectiveness  criteria  identified 
in  the  research  effort  of  Bradney  and  Perkins  (6:34).  These 
researchers  suggested  that  the  identified  measurement  criteria 
could  be  incorporated  into  an  evaluation  plan  for  the  RIW 
concept.  The  criteria  identified  by  Bradney  and  Perkins, 
and  the  associated  definitions  as  applied  to  this  research 
effort,  are  found  in  Table  3-1.  Responses  to  this  section 
were  used  to  answer  Research  Question  One. 

The  third  section,  Future  Recommendation,  indicated 
the  recommendations  for  future  use  by  the  Air  Force  of  the 
RIW  concept,  as  perceived  by  the  RIW  experts.  Responses  to 
this  section  were  used  to  answer  Research  Question  Two. 

The  fourth  and  final  section  of  the  interview  instru¬ 
ment,  Open-Ended  Questions,  consisted  of  two  questions  concern¬ 
ing  significant  subjects  of  interest  to  this  research  team. 

The  questions  permitted  the  RIW  experts  to  respond  to  questions 
without  the  structure  imposed  by  the  second  section.  The 
research  team  reviewed  and  interpreted  these  open-ended 
questions  to  ascertain  RIW  experts'  feelings  and  expressions 
of  intensity  about  the  RIW  concept  (9:234).  Furthermore, 
responses  to  these  questions  were  compared  to  the  results 
obtained  in  answering  the  two  Research  Questions. 


TABLE  3-1 


EFFECTIVENESS  CRITERIA  DEFINED 


Reliability  -  The  probability  that  an  item  will  perform  its 
intended  function  for  a  specified  interval  under  stated 
conditions . 

Maintainability  -  The  probability  that  an  item  will  be 

restored  to  a  specified  condition  within  a  given  period 
of  time  when  proper  maintenance  is  performed. 

Acquisition  Cost  -  The  total  expenditures  incurred  for  the 
purchase  or  production  of  an  item. 

Equipment  Modification  (ECN)  -  Alterations  in  the  physical 
or  functional  characteristics  of  an  item  after  initial 
establishment  of  these  characteristics. 

Data  Collection  -  Collection  of  required  data  (R&D ,  produc¬ 
tion,  training,  modification,  overhaul,  O&M,  etc.) 
during  item  procurement  and  support  by  all  levels  of 
management . 

Contractor  Competition  -  Competition  between  responsible, 
responsive  contractors  during  all  phases  of  item 
acquisition  and  support. 

Use  Rate  -  The  quantity  of  usage  by  type  that  occurs  in  a 
specific  time  period. 

Supply  System  -  Organizations,  offices,  facilities,  methods, 
and  techniques  necessary  to  provide  an  item  to 
authorized  users. 

Maintenance  Skill  -  Skill  required  at  all  maintenance  levels 
to  provide  proper  and  adequate  item  repair. 

Number  of  Spares  -  Articles  identical  to  or  interchangeable 
with  the  end  item  which  are  procured  over  and  above 
the  quantity  needed  for  initial  installation. 

RIW  Contract  Clause  -  Content  necessary  to  insure  that 

required  acquisition  and  support  goals  and  objectives 
are  established  for  procurement  of  an  item. 


Testing 


Emory  states  that  "...  once  a  first  draft  of  the 
instrument  has  been  developed  it  must  be  tested  [9:222]." 

The  purpose  of  this  testing  was  to  insure  that  the  instru¬ 
ment  used  was  understandable,  unambiguous,  clear  to  both 
interviewer  and  interviewee,  and  concise.  The  research 
team  used  several  Air  Force  officers  and  civilian  personnel 
with  similar  backgrounds  to  the  RIW  experts  to  act  as  inter¬ 
viewees  in  order  to  evaluate  and  provide  feedback  on  the 
interview  instrument.  This  testing  was  iterative  until  only 
minor  problems  requiring  revision  surfaced. 

The  Measurement  Scales 

It  was  important  to  be  able  to  quantify  the  RIW 
experts'  responses  in  order  to  answer  the  research  questions. 
The  measurement  scales  used  in  the  interview  instrument 
were  both  a  seven-point  and  a  three-point  Likert  scale.  The 
scale  for  the  Program  Evaluation  section  consisted  of  the 
following  seven  possible  responses : 

1.  Least 

2.  Less 

3.  Slightly  Less 

4 .  Same  As 

5.  Slightly  More 

6 .  More 

7.  Most 
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The  scale  for  the  Future  Recommendation  section  consisted 
of  the  following  three  possible  responses: 

1.  Discontinue 

2 .  Undecided 

3.  Continue 

The  Likert  scale  was  selected  because  of  its  ease 
of  construction  and  discriminating  ability  in  respondent- 
centered  research  (9:273).  Emory  states: 

...  a  two-point  scale,  three-point  scale, 
or  scales  with  more  points  is  a  subject  which  is 
debated;  there  is  little  conclusive  support  for 
any  particular  scale  length.  One  argument  is  that 
more  points  on  a  scale  provide  an  opportunity  for 
greater  sensitivity  of  measurement.  However,  the 
most  widely  used  scales  range  from  three  to  seven 
points,  and  it  does  not  seem  to  make  much  difference 
which  number  is  used  [9:261]. 

As  a  result,  the  research  team  determined  that  both  a  seven- 
point  and  three-point  scale  best  satisfied  the  requirements 
of  this  research  effort. 

The  Likert  scales  used  resulted  in  ordinal  level  data 
(9:274).  Ordinal  level  data  is  defined  as  data  which  are 
ranked  (least  to  most)  with  respect  to  a  measured  attribute, 
but  the  distance  between  the  ranked  items  is  unknown  (23:23-24) 
According  to  Emory,  ".  .  .we  can  report  respondents  are  more 
or  less  favorable  to  a  topic,  but  we  cannot  tell  how  much 
more  or  less  favorable  they  are  [9:274]."  The  primary  attri¬ 
bute  measured  in  this  research  effort  was  RIW  effectiveness. 
Thus,  this  research  team  reported  whether  or  not  RIWs  had 
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been  more  or  less  effective,  but  could  not  report,  to  any 
degree,  how  much  more  or  less  effective  RIWs  had  been. 

Data  Analysis  Plan 

As  discussed  previously,  the  data  in  the  Program 
Evaluation  and  Future  Recommendation  sections  of  the  inter¬ 
view  instrument  were  ordinal  level  data.  As  such,  nonpara- 
metric  statistical  tests  were  used  to  perform  the  statistical 
analysis.  Emory  (9:413)  suggests  that  nonparametric  tests 
".  .  .  are  the  only  technically  correct  tests  to  use  with 
ordinal  data."  Additionally,  Harnett  says, 

...  to  avoid  the  parametric  assumptions 
normally  required  for  tests  based  on  ratio  and 
interval  scales,  most  nonparametric  tests  assume 
only  nominal  or  ordinal  data  [12:695]. 

For  these  reasons,  statistical  analysis  of  the  Program 
Evaluation  and  Future  Recommendation  sections  were  justified 
using  nonparametric  tests.  Analysis  of  the  General/Demo¬ 
graphic  information  of  the  interview  instrument  was  descrip¬ 
tive  in  nature  and  presented  in  tabular  form.  In  the  last 
section  of  the  interview  instrument,  analysis  of  the  open- 
ended  questions  consisted  of  a  qualitative  evaluation  by  the 
research  team  to  compare  these  responses  to  the  information 
in  the  Program  Evaluation  and  Future  Recommendation  sections. 
The  research  questions  and  supporting  qualitative  research 
hypotheses  that  comprised  this  research  effort’s  methodology 


are  restated: 


Research  Question  One.  What  are  the  perceptions  of 
active  duty  Air  Force  officers  and  DOD  civilians  with  Air 
Force  RIW  expertise  regarding  the  effectiveness  of  RIWs  as 
applied  to  Air  Force  programs  since  August  1974? 

Supporting  Qualitative  Research  (Null)  Hypotheses 
for  Research  Question  One. 

1-6.  Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  perceive  the 

(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 

RIW  program  as  ineffective  (this  represents  one  hypothesis 
for  each  of  the  six  Air  Force  RIW  programs) . 

7.  Active  duty  Air  Force  officers  and  DOD 
civilians  with  Air  Force  RIW  expertise  perceive  the  Air  Force 
RIW  program  as  ineffective. 

Research  Question  Two.  What  is  the  recommendation 
for  the  future  use  by  the  Air  Force  of  the  RIW  concept,  as 
perceived  by  Air  Force  officers  and  DOD  civilians  with  Air 
Force  RIW  expertise? 
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orting  Qualitative  Research  (.Null)  Hypothesis 
for  Research  Question  Two. 

8.  Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  do  not  recommend  the  continued 
use  by  the  Air  Force  of  the  RIW  concept. 

Statistical  Methodology 

The  most  appropriate  statistic  of  central  tendency 
in  an  ordinal  scale  is  the  median,  since  the  median  is  .  . 
not  affected  by  changes  of  any  scores  which  are  above  or 
below  it  as  long  as  the  number  of  scores  above  and  below 
remains  the  same  [23:25]."  Harnett  defines  the  median, 

" .  .  .as  the  middle  value  in  a  set  of  numbers  arranged  in 
order  of  magnitude  [12:15]."  If  the  total  number  of  values 
in  the  data  set  is  odd ,  the  median  is  the  middle  of  the  data 
set  (12:15).  If  the  total  number  of  values  in  the  data 
set  is  even,  the  median  is  the  average  of  the  two  middle 
values  of  the  data  set  (12:15).  Throughout  this  research 
effort,  the  median  was  used  to  evaluate  the  research  questions 
and  hypotheses  presented  earlier. 

To  use  the  median,  the  arabic  numerical  scales  (i.e., 
1-7  and  1-3  inclusive) ,  associated  with  the  two  Likert  scales 
in  the  interview  instrument  was  used.  The  purpose  of  these 
arabic  numerical  scales  was  to  calculate  the  numerical 


medians . 


The  technique  used  to  evaluate  (measure)  perceived 
effectiveness  of  the  Air  Force  RIW  program  and  to  evaluate  the 
perceived  recommendation  for  future  use  by  the  Air  Force  of 
the  RIW  concept  involved  the  RIW  expert's  individual  ratings. 
Research  Hypothesis  One  through  Six  was  evaluated  using  the 
set  of  RIW  expert's  individual  ratings  for  each  Air  Force 
RIW  program;  and  Research  Hypotheses  Seven  and  Eight  were 
evaluated  using  the  set  of  all  the  RIW  expert's  individual 
ratings . 

Prior  to  conducting  nonparametric  tests  associated 
with  Research  Hypotheses  One  through  Eight  to  evaluate  the 
two  research  questions,  the  use  of  contingency  table  analysis 
or  other  validating  means  was  not  required  to  validate  the 
RIW  experts'  effectiveness  criteria  ratings.  In  this 
research  effort,  the  RIW  experts'  ratings  were  assumed  a 
valid  measure  of  perceived  effectiveness  of  Air  Force  RIWs. 
While  validation  of  RIW  experts'  ratinys  was  not  required, 
further  definition  of  these  ratings  was  required  before 
statistical  analysis. 

To  further  define  the  RIW  experts'  effectiveness 
criteria  ratings,  a  rating  of  four  (4)  on  the  seven-point 
scale,  previously  discussed,  was  selected  as  the  neutral 
rating.  This  neutral  rating  is  the  division  between  an 
effective  and  ineffective  rating.  Therefore,  for  the  effec¬ 
tiveness  criteria  of  Reliability,  Maintainability,  Contractor 


Competition,  Use  Rate  or  RIW  Contract  Clause  a  rating 
greater  than  four  indicated  RIW  effectiveness;  whereas,  a 
rating  of  less  than  or  equal  to  four  indicated  RIW  effective¬ 
ness.  For  the  effectiveness  criteria  of  Acquisition  Cost, 
Equipment  Modification,  Data  Collection,  Supply  System, 
Maintenance  Skill  or  Number  of  Spares,  a  rating  of  less  than 
four  indicated  RIW  effectiveness;  whereas,  a  rating  of 
greater  than  or  equal  to  four  indicated  RIW  ineffectiveness. 
Table  3-2  illustrates  the  definition  of  RIW  experts'  ratings. 
To  facilitate  statistical  analysis,  the  RIW  experts'  ratings 
of  the  effectiveness  criteria  of  Acquisition  Cost,  Equipment 
Modification,  Data  Collection,  Supply  System,  Maintenance 
Skill  and  Number  of  Spares  was  recoded  to  insure  consistency 
of  experts'  ratings.  For  example,  if  an  RIW  expert  rated 
Acquisition  Cost  as  two  (2) ,  this  criterion  was  recoded  as 
six  (6) ,  which  implies  the  same  level  of  effectiveness. 
Therefore,  after  recoding,  for  all  eleven  effectiveness 
criteria  a  rating  greater  than  four  indicated  RIW  effective¬ 
ness;  whereas,  a  rating  less  than  or  equal  to  four  Indicated 
RIW  ineffectiveness.  Table  3-3  illustrates  the  recoding  of 
RIW  experts'  ratings. 

Following  further  definition  and  recoding  of  the  RIW 


experts'  criteria  ratings,  the  eight  supporting  qualitative 
research  hypotheses  used  in  conjunction  with  the  two  research 
questions  were  individually  evaluated.  In  evaluating  these 
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hypotheses,  the  following  five-step  procedure  of  hypothesis 
testing  (12:351)  was  considered: 

1.  State  the  null  (Hq)  and  alternative  (HA)  hypo¬ 
theses; 

2.  Determine  the  appropriate  test  statistic; 

3.  Determine  the  critical  region; 

4.  Compute  the  value  of  the  test  statistic;  and 

5.  Make  the  statistical  decision  and  interpretation. 
The  statistical  eauivalent  and  for  each  of 

the  Research  Hypotheses  One  through  Six  was: 

Hq:  The  median  RIW  experts'  individual  ratings  on 
all  eleven  effectiveness  criteria  on  the 

(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 

RIW  programs  <  4  (i.e.,  the  neutral  rating) 

Ha:  The  median  RIW  experts'  individual  ratings  on 
all  eleven  effectiveness  criteria  on  the 

(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 


(4) 


Carousel  IV  INS 


a 


(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 
RIW  program  >  4 . 

=  .05 

By  rejecting  any  of  the  six  Hq's,  the  research  team  would  have 
concluded  that  the  RIW  experts  perceive  the  RIW,  with  respect 
to  that  Air  Force  program  as  effective.  By  accepting  any 
of  the  six  H0's,  the  research  team  would  have  concluded  that 
the  RIW  experts  perceive  the  RIW,  with  respect  to  that  Air 
Force  RIW  program,  as  ineffective. 

After  Research  Hypotheses  One  through  Six  were  tested 
to  determine  individual  program  effectiveness  or  ineffective¬ 
ness,  two  statistical  tests  were  developed  to  test  Research 
Hypothesis  Seven.  First,  a  statistical  test  was  developed  to 
simply  determine  whether  a  significant  percentage  of  the  Air 
Force  RIW  programs  were  effective.  The  statistical  equivalent 
Hq  and  Ha  for  the  first  test  of  Research  Hypothesis  Seven  was: 

Hq:  The  proportion  of  effective  Air  Force  AIR 
programs  £0.5 

Ha:  The  proportion  of  effective  Air  Force  RIW 
programs  >  0.5 
a  =  .15 

By  rejecting  Hq,  the  research  team  would  have  concluded  that 
the  RIW  experts  perceive  the  overall  Air  Force  RIW  program  as 
effective.  By  accepting  H  ,  the  research  team  would  have 


concluded  that  the  RIW  experts  perceive  the  overall  Air  Force 
RIW  program  as  ineffective.  This  first  statistical  test 
alone  would  be  somewhat  restrictive  with  a  =  .05  level,  all 
six  Air  Force  RIW  programs  would  have  to  be  effective  (i.e., 
using  the  Binomial  Test  with  proportion  =  0.5,  the  proba¬ 
bility  of  six  effective  programs  is  .016  and  the  probability 
of  at  least  five  effective  programs  is  .11;  therefore,  at 
a  =  .05,  Hq  would  only  be  rejected  for  six  effective  programs) 
To  be  less  restrictive,  the  significance  level  was,  therefore, 
reduced  to  a  =  .15,  thus  requiring  only  five  of  the  six  Air 
Force  RIW  programs  to  be  effective  before  HQ  is  rejected. 

To  compensate  for  the  reduced  significance  level ,  a  second 
statistical  test  was  constructed  using  the  set  of  all  the 
RIW  experts'  individual  ratings.  The  statistical  equivalent 
Hq  and  Ha  for  the  second  test  of  Research  Hypothesis  Seven 
was : 

Hq:  The  median  RIW  experts'  individual  ratings  on  all 
eleven  effectiveness  criteria  on  all  six  Air  Force  RIW 
programs  <_  4 . 

Ha:  The  median  RIW  experts'  individual  ratings  on  all 
eleven  effectiveness  criteria  on  all  six  Air  Force  RIW 
programs  >  4 . 

a  =  .  05 

By  rejecting  HQ,  the  research  team  would  have  concluded  that 
the  RIW  experts  perceive  the  overall  Air  Force  RIW  program  as 


effective.  By  accepting  HQ,  the  research  team  would  have 
concluded  that  the  RIW  experts  perceive  the  overall  Air 
Force  program  as  ineffective..  Finally,  by  rejecting 
hq  on  both  statistical  tests  for  Research  Hypothesis  Seven, 
the  research  team  would  have  concluded  that  the  RIW  experts 
perceive  the  overall  Air  Force  RIW  program  as  effective. 
Therefore,  by  accepting  HQ  on  one  or  both  statistical  tests 
for  Research  Hypothesis  Seven,  the  research  team  would  have 
concluded  that  the  RIW  experts  perceive  the  overall  Air 
Force  program  as  ineffective. 

The  null  and  alternative  hypotheses  for  Research 
Question  Two  were  developed  in  two  phases,  with  phase  two 
dependent  on  the  results  in  phase  one.  The  statistical 
equivalent  HQ  and  HA  for  phase  one  of  Research  Hypothesis 
Eight  was: 

HQ:  The  median  RIW  experts'  ratings  on  the  future 
use  by  the  Air  Force  of  the  RIW  concept  =  2. 

Ha:  The  median  RIW  experts'  ratings  on  the  future 

use  by  the  Air  Force  of  the  RIW  concept  is  ^  2. 

a  =  .05 

By  accepting  HQ,  the  research  team  would  have  concluded  that 
the  RIW  experts  are  undecided  on  the  future  use  by  the  Air 
Force  of  the  RIW  concept.  Therefore,  phase  two  of  Research 
Hypothesis  Eight  would  not  be  required.  By  rejecting  HQ,  th 
research  team  would  have  concluded  that  the  RIW  experts  are 


decided  on  the  future  use  by  the  Air  Force  of  the  RIW 
concept.  Therefore,  phase  two  of  Research  Hypothesis  Eight 
would  be  required  to  determine  the  RIW  experts '  recommenda¬ 
tion  on  the  future  use  by  the  Air  Force  of  the  RIW  concept. 

The  RIW  experts'  recommendation  was  determined  by  the  majority 
of  positive  or  negative  signed  ranks  of  the  statistical  test. 
The  sign  and  quantity  of  the  ranks  of  the  statistical  test 
indicated  on  which  side  of  the  median  the  majority  of  the 
distribution  lies.  If  the  signed  ranks  are  positive  (+) , 
this  indicates  the  RIW  experts'  ratings  are  greater  than 
the  median  (2) ;  and  if  the  signed  ranks  are  negative  (-) , 
this  indicates  the  RIW  experts'  ratings  are  less  than  the 
median  (2) .  Therefore,  if  the  majority  of  signed  ranks  are 
positive,  the  research  team  would  have  concluded  that  the 
RIW  experts  recommend  continued  use  by  the  Air  Force  of  the 
RIW  concept;  and  if  the  majority  of  signed  ranks  are  negative, 
the  research  team  would  have  concluded  that  the  RIW  experts 
do  not  recommend  continued  use  by  the  Air  Force  of  the  RIW 
concept. 

The  statistical  test  used  for  Research  Hypotheses 
One  through  Six  was  the  Wilcoxon  Matched-Pairs  Signed-Ranks 
Test  (23:75-83).  This  statistical  test  enabled  the  research 
team  to  compare  RIW  experts '  individual  ratings  to  a  neutral 
rating  to  determine  whether  the  RIW  experts  perceive  each  of 
the  six  Air  Force  programs  to  be  effective.  The  statistical 
tests  used  for  Research  Hypothesis  Seven  were  a  Binomial 


Test  (23:36-42)  and  the  Wilcoxon  Matched-Pairs  Signed-Ranks 
Test  (23:75-83).  These  statistical  tests  enabled  the 
research  team  to  determine  the  effectiveness  of  the  overall 
Air  Force  RIW  program  as  perceived  by  the  RIW  experts.  The 
statistical  test  used  for  Research  Hypothesis  Eight  was  the 
Wilcoxon  Matched-Pairs  Signed-Ranks  Test  (23:75-83).  This 
statistical  test  enabled  the  research  team  to  compare  RIW 
experts'  individual  ratings  to  a  neutral  rating  to  determine 
the  RIW  experts '  recommendation  for  the  future  use  by  the 
Air  Force  of  the  RIW  concept. 

Assumptions  and  Limitations 

In  this  research  project,  this  research  team  made 
several  assumptions  and  identified  the  scope  and  limitations. 
These  assumptions  were  categorized  as  statistical  assumptions 
and  general  assumptions.  Both  categories  of  assumptions  and 
the  identified  scope  and  limitations  were  necessary  for 
research  consistency,  and  are  detailed  below. 

Statistical  Assumptions 

1.  The  use  of  the  seven-point  and  three-point 
Likert  scales  in  the  interview  instrument  resulted  in  ordinal 
level  data. 

2.  The  use  of  nonparametric  statistics  was  justified 
due  to  the  unknown  population  distributions  from  which  the 
sample  was  drawn. 
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3.  Nonpararaetric  techniques  provided  statistical 


analysis  of  ordinal  level  data. 

4.  The  purposive  sampling  method  was  a  valid 
technique  for  developing  a  sample  for  nonparametric  testing. 

General  Assumptions 

1.  Each  RIW  expert's  effectiveness  criteria  ratings 
were  a  valid  measure  of  perceived  effectiveness  of  Air  Force 
RIWs . 

2.  The  RIW  experts  provided  honest  responses  and 
took  a  reasonable  amount  of  time  to  consider  responses  to 
each  criterion  of  the  interview  instrument. 

3.  The  interview  instrument  was  a  reliable  attitude 
measurement  tool  that  was  adequately  tested. 

4.  The  research  team's  definition  of  effectiveness 
was  appropriate  for  this  research  effort. 

5.  The  RIW  effectiveness  criteria  identified  by  pre 
vious  research  was  appropriate  and  valid  for  this  research 
effort. 

6.  Two  years  in  program  management,  contracting 
management,  or  logistics  management,  on  any  of  the  six  Air 
Force  RIW  programs,  yielded  the  experience  necessary  to 
consider  individuals  knowledgeable  of  Air  Force  RIWs. 

Scope  and  Limitations 

1.  Individuals  with  Air  Force  RIW  expertise  were 
limited  to  active  duty  Air  Force  officers  and  DOD  civilians. 


2.  Only  those  Air  Force  RIW  programs  previously 
identified  were  included  in  this  research  effort. 

3.  The  conclusions  of  this  research  effort  applied  . 
only  to  the  RIW  experts  identified  earlier  in  this  chapter. 

4.  The  measurement  of  RIW  experts'  perceptions  were 
qualitative  in  nature. 

5.  The  answers  to  the  open-ended  questions  were 
reviewed  and  interpreted  by  the  research  team  and  conclusions 
were  based  on  the  team's  subjective  judgment. 


Summary 

Chapter  I  described  the  need  for  determining  RIW 
effectiveness,  defined  the  research  problem,  and  identified 
research  objectives.  Chapter  II  provided  background  and 
discussed  the  origins  of  Reliability  Improvement  Warranties, 
the  use  of  RIWs  by  DOD,  and  the  development  of  the  Air  Force 
RIW  program  from  its  inception  to  the  present  day.  This 
chapter  described  the  methodology  to  be  used  by  the  research 
team  to  evaluate  the  effectiveness  of  the  Air  Force  RIW 
program.  Using  this  methodology,  the  research  team  analyzed 
the  data  in  Chapter  IV. 


CHAPTER  IV 


DATA  ANALYSIS 

Introduction 

The  purpose  of  the  Data  Analysis  Chapter  is  to  pre¬ 
sent  and  analyze  the  interview  data  using  the  research 
methodology  developed  in  Chapter  III.  This  chapter  includes 
(1)  a  descriptive  tabular  analysis  of  the  General /Demographic 
information,  (2)  analysis  of  the  two  research  questions  pre¬ 
sented  in  Chapter  I,  and  (3)  a  qualitative  evaluation  of  the 
open-ended  interview  questions.  Analysis  of  the  two  research 
questions  consists  of  nonparametric  statistical  tests. 

General /Demographic  Information 

The  RIW  experts,  as  identified  in  Chapter  III,  were 
those  individuals  who 

1.  have  a  basic  knowledge  of  the  problem  area  and 
are  able  to  apply  that  knowledge; 

2.  have  experience  formulating,  implementing  or 
monitoring  an  Air  Force  RIW  program;  and 

3.  have  two  or  more  years  experience  in  program 
management,  contract  management,  or  logistics  management  on 
any  of  the  six  Air  Force  RIW  programs. 

The  following  tables  describe  the  general  charac¬ 
teristics  of  the  RIW  experts.  Inferences  concerning  the  RIW 
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experts  are  neither  implied  nor  attempted  for  the  entire 
population  of  active  duty  Air  Force  officers  and  DOD  civilians 
previously  or  currently  involved  in  one  of  the  six  Air  Force 
RIW  programs. 

Table  4-1  illustrates  Air  Force  officer  and  DOD 
civilian  RIW  experts  by  general  career  field.  A  majority 
(21  of  36  or  58.3  percent)  of  the  RIW  experts  have  logistics 
management  career  field  experience.  The  RIW  experts  with 


program  management  (10  of  36)  and  contract  management 
(5  of  36)  experience  constitute  27.8  percent  and  13.9  per¬ 
cent,  respectively. 


TABLE  4-1 

RIW  EXPERTS  BY  GENERAL  CAREER  FIELD 


General  Career 
Field 

Air  Force 
Officers 

Per¬ 

cent 

Per¬ 

cent 

Total 

Program 

Management 

7 

100 

3 

10.3 

10 

Contract 

Management 

0 

0 

5 

17.2 

5 

Logistics 

Management 

0 

0 

21 

72.5 

21 

Total 

7 

100 

29 

100.0 

36 

TABLE  4-2 

RIW  PROGRAM  BY  GENERAL  CAREER  FIELD 


General  Career 
Field 

TACAN 

AHRS 

OMEGA 

C-IV 

F-16 

AF  St 

INU 

Program 

Management 

2 

2 

1 

1 

2 

2 

Contract 

Management 

1 

0 

3 

0 

1 

0 

Logistics 

Management 

3 

3 

3  1 

4 

4 

4 

Total 

6 

5 

7 

5 

7 

6 

Table  4-3  illustrates  the  years  of  experience  by 
general  career  field.  The  overall  average  years  of  experience 
of  the  RIW  experts  is  16.4  years. 


TABLE  4-3  ' 

YEARS  OF  EXPERIENCE  BY  GENERAL 
CAREER  FIELD 


General  Career  Field 

Cumulative 

Yrs  Experience 

Average 

Yrs  Experience 

Program  Management 

87 

8.7 

Contract  Management 

88 

17.6 

Logistics  Management 

414 

19.7 

Total 

589 

16.4 
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Table  4-4  lists  the  RIW  experts  by  specific  Air 
Force  Specialty  Code  (AFSC) /Civilian  Job  Series.  A  plurality 
(38.9  percent)  of  the  RIW  experts  possess  a  346  job  series 
(Logistics  Management  Specialist) . 


TABLE  4-4 

RIW  EXPERTS  BY  SPECIFIC 
AFSC /JOB  SERIES 


AFSC/Job  Series 

Program 

Mgmt 

Contract 

Mgmt 

* 

Logistics 

Mgmt 

Percent 

301 

1 

2.8 

346 

14 

38.9 

800 

2 

3 

13.8 

856 

1 

2.8 

1102 

4 

11.1 

1105 

1 

2.8 

1670 

2 

5.6 

27XX 

4 

11.1 

28XX 

2 

1 

'.3 

66XX 

1 

2.8 

Total 

10 

5 

21 

100.0 

Table  4-5  illustrates  the  years  of  RIW  experience  by 
general  career  field.  The  overall  average  years  of  RIW 
experience  of  the  RIW  experts  (152  cumulative  years  of  RIW 
experience  divided  by  the  36  RIW  experts)  is  4.2  years. 
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TABLE  4-5 


YEARS  OF  RIW  EXPERIENCE  BY 
GENERAL  CAREER  FIELD 


General  Career  Field 

Cumulative 

Yrs  Experience 

Average 

Yrs  Experience 

Program  Management 

33 

3.3 

Contract  Management 

22 

4.4 

Logistics  Management 

97 

4.6 

Total 

152 

4.2 

Table  4-6  lists  years  of  RIW  experience  per  RIW 
expert  for  each  of  the  six  Air  Force  RIW  programs..  As  an 
example,  the  third  RIW  expert  on  the  C-141  AHRS  program  had 
five  years  of  RIW  experience,  whereas  the  fourth  RIW  expert 
on  that  program  had  two  years  of  RIW  experience.  Of  the  RIW 
experts  interviewed,  the  greatest  number  of  years  of  RIW 
experience  is  on  the  F-16  program. 


TABLE  4-6 

YEARS  OF  RIW  EXPERIENCE  BY  PROGRAM 


Program 

Years 

of  RIW  Experience 

Per  Expert 

Cumulative 
Years  of  RIW 
Experience 

■I 

2 

3 

4 

5 

6 

m 

ARN  118  TACAN 

2 

5 

5 

6 

5 

6 

- 

29 

C-141  AHRS 

5 

3 

5 

2 

3 

- 

- 

18 

OMEGA  Nav  Set 

5 

5 

3 

3 

3 

4 

3 

26 

C-IV  INS 

5 

5 

4 

4 

4 

- 

- 

22 

F-16  Elex 

2 

3 

7 

4 

7 

6 

7 

36 

AF  St  INU 

5 

3 

3 

3 

4 

5 

- 

23 

Program  Evaluation 

Research  Question  One 

What  are  the  perceptions  of  active  duty  Air  Force 
officers  and  DOD  civilians  with  Air  Force  RIW  expertise 
regarding  the  effectiveness  of  RIWs  applied  to  Air  Force 
programs  since  1974?  Seven  supporting  qualitative  research 
hypotheses  were  used  to  answer  Research  Question  One: 

1 .  Research  (Null)  Hypotheses  One  through  Six: 

Active  duty  Air  Force  officers  and  DOD  civilians 
with  Air  Force  RIW  expertise  perceive  the 

(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 
RIW  program  as  ineffective. 

a.  Statistical  Test:  Wilcoxon  Matched-Pairs 

Signed-Ranks  Test,  a  =  .05.  The  statistical 
equivalent  HQ  and  for  each  of  the  Research 
Hypotheses  One  through  Six  are: 


Hq:  The  median  RIW  experts'  individual 


ratings  on  all  eleven  effectiveness 
criteria  on  the 


(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA -Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 

RIW  program  <_  4 . 

Ha:  The  median  RIW  experts'  individual 

ratings  on  all. eleven  effectiveness 

criteria  on  the 

(1)  AN/ARN  118  TACAN 

(2)  C-141  AHRS 

(3)  OMEGA  Nav  Set 

(4)  Carousel  IV  INS 

(5)  F-16  Electronic  Components 

(6)  Air  Force  Standard  INU 

RIW  program  >  4 . 

Because  this  was  a  one-tailed  statistical 
test,  the  decision  to  reject  HQ  was  deter¬ 
mined  if  one-half  the  calculated  two-tailed 
probability  was  less  than  the  significance 
level  chosen,  ct  =  .05.  The  null  hypothesis 
is  rejected  if  the  calculated  z-statistic 
has  a  probability  less  than  the  specified 
alpha  level  (15:228). 


Results ;  Appendix  C,  Tables  C-l  through 
C-6  illustrate  results  of  the  Wilcoxon 
Matched-Pairs  Signed-Ranks  Test  for  the  RIW 
experts  on  each  of  the  six  Air  Force  RIW 
programs.  On  the  AN/ARN  118  TACAN  RIW 
program,  one-half  the  calculated  two-tailed 
probability  was  .0155,  which  was  less  than 
the  significance  level  of  a  =  .05.  There¬ 
fore,  the  research  team  rejected  HQ  and 
were  95  percent  confident  that  the  RIW 
experts  perceived  the  AN/ARN  118  TACAN  RIW 
program  as  effective.  On  the  C-141  AHRS 
RIW  program,  one-half  the  calculated  two- 
tailed  probability  was  .0005,  which  was  less 
than  the  significance  level  of  a  =  .05. 
Therefore,  the  research  team  rejected  HQ 
and  were  95  percent  confident  that  the  RIW 
experts  perceived  the  C-141  AHRS  RIW  program 
as  effective .  On  the  OMEGA  Nav  Set  RIW 
program,  one-half  the  calculated  two-tailed 
probability  was  .0165,  which  was  less  than 
the  significance  level  of  a  =  .05.  Therefore 
the  research  team  rejected  HQ  and  were  95 
percent  confident  that  the  RIW  experts  per¬ 
ceived  the  OMEGA  Nav  Set  RIW  program  as 


effective.  On  the  Carousel  IV  INS  RIW 
program,  one-half  the  calculated  two-tailed 
probability  was  .0055,  which  was  less  than 
the  significance  level  of  a  =  .05.  There¬ 
fore,  the  research  team  rejected  and  were 
95  percent  confident  that  the  RIW  experts 
perceived  the  Carousel  IV  INS  RIW  program  as 
effective.  On  the  F-16  Electronic  Components 
RIW  program,  one-half  the  calculated  two- 
tailed  probability  was  .005,  which  was  less 
than  the  significance  level  of  a  =  .05. 
Therefore,  the  research  team  rejected  HQ 
and  were  95  percent  confident  that  the  RIW 
experts  perceived  the  F-16  Electronic 
Components  program  as  effective.  On  the  Air 
Force  Standard  INU  RIW  program,  one -half  the 
calculated  two-tailed  probability  was  .0005, 
which  was  less  than  the  significance  level 
of  a  =  .05.  Therefore,  the  research  team 
rejected  HQ  and  were  95  percent  confident 
that  the  RIW  experts  perceived  the  Air  Force 
Standard  INU  program  as  effective. 

Research  (Null)  Hypothesis  Seven:  Active  duty 
Air  Force  officers  and  DOD  civilians  with  Air 
Force  RIW  expertise  perceive  the  Air  Force  RIW 
program  as  ineffective. 


Statistical  Test  One:  Binomial  Test, 
a  =  .15.  The  statistical  HQ  and  HA  for  the 
first  test  of  Research  Hypothesis  Seven  are-: 

Hq :  The  proportion  of  effective  Air  Force 

RIW  programs  <_  .05. 

Ha:  The  proportion  of  effective  Air  Force 
RIW  programs  >  .05. 

Since  this  was  a  one-tailed  statistical 
test,  the  decision  to  reject  HQ  was  deter¬ 
mined  if  the  calculated  one-tailed  proba¬ 
bility  was  less  than  the  significance  level 
chosen,  a  =  .15. 

Statistical  Test  Two:  Wilcoxon  Matched-Pairs 
Signed-Ranks  Test,  a  =  .05.  The  statistical 
equivalent  HQ  and  HA  for  the  second  test  of 
Research  Hypothesis  Seven  are: 

Hq:  The  median  RIW  experts'  individual 
ratings  on  all  eleven  effectiveness 
criteria  on  all  six  Air  Force  RIW 
programs  _<  4 . 

Ha:  The  median  RIW  experts'  individual 
ratings  on  all  eleven  effectiveness 
criteria  on  all  six  Air  Force  RIW 
programs  >  4 . 

Because  this  was  a  one-tailed  statistical 
test,  the  decision  to  reject  H^  was  determined 


if  one-half  the  calculated  two-tailed 
probability  was  less  than  the  significance 
level  chosen,  a  =  .05. 

Results :  Appendix  C,  Tables  C-7  and  C-8 
illustrate  the  results  of  the  Binomial  Test 
and  the  Wilcoxon  Matched-Pairs  Signed-Ranks 
Test  for  the  RIW  experts  on  all  six  of  the 
Air  Force  RIW  programs.  From  the  Binomial 
Test,  the  calculated  one-tailed  probability 
was  .016,  which  was  less  than  the  significance 
level  of  a  =  .15.  Therefore,  the  research 
team  rejected  the  first  HQ  and  were  85  per¬ 
cent  confident  that  the  RIW  experts  per¬ 
ceived  the  overall  Air  Force  RIW  program  as 
effective.  From  the  Wilcoxon  Test,  one-half 
the  calculated  two-tailed  probability  was 
.000,  which  was  less  than  the  significance 
level  of  a  =  .05.  Therefore,  the  research 
team  rejected  the  second  HQ  and  were  95  per¬ 
cent  confident  that  the  RIW  experts  perceived 
the  overall  Air  Force  RIW  program  as  effec¬ 
tive.  By  rejecting  HQ  on  both  statistical 
tests  for  Research  Hypothesis  Seven,  the 
research  team  concluded  that  the  RIW  experts 
perceive  the  overall  Air  Force  RIW  program  as 


effective . 


Future  Recommendation 


Research  Question  Two 

What  is  the  recommendation  for  the  future  use  by 
the  Air  Force  of  the  RIW  concept,  as  perceived  by  Air  Force 
officers  and  DOD  civilians  with  Air  Force  RIW  expertise? 

One  supporting  qualitative  research  hypothesis  was  used  to 
answer  Research  Question  Two: 

1.  Research  (Null)  Hypothesis  Eight:  Active  duty 
Air  Force  officers  and  DOD  civilians  with  Air 
Force  RIW  expertise  do  not  recommend  the  con¬ 
tinued  use  by  the  Air  Force  of  the  RIW  concept, 
a.  Statistical  Test:  Wilcoxon  Matched-Pairs 

Signed-Ranks  Test,  a  =  .05.  The  statistical 
equivalent  HQ  and  HA  for  phase  one  of 
Research  Hypothesis  Eight  are: 

Hq:  The  median  RIW  experts'  ratings  on 
the  future  use  by  the  Air  Force  of 
the  RIW  concept  =  2. 

Ha:  The  median  RIW  experts'  ratings  on  the 
future  use  by  the  Air  Force  of  the 
RIW  concept  ^  2. 

Since  this  is  a  two-tailed  statistical  test, 
the  decision  to  reject  HQ  was  determined  if 
the  calculated  two-tailed  probability  was 
less  than  the  significance  level  chosen, 


b.  Results ;  Appendix  C,  Table  C-9  illustrates 
results  of  the  Wilcoxon  Matched-Pairs  Signed 
Ranks  Test  for  the  RIW  experts  on  the  future 
use  by  the  Air  Force  of  the  RIW  concept. 

The  calculated  two-tailed  probability  was 
.000,  which  was  less  than  the  significance 
level  of  a  =  .05.  Therefore,  the  research 
team  rejected  HQ  and  were  95  percent  confi¬ 
dent  that  the  RIW  experts  were  decided  on 
the  future  use  by  the  Air  Force  of  the  RIW 
concept.  Appendix  C,  Table  C-9  illustrates 
that  the  majority  of  the  signed  ranks  are 
positive  (30  +Ranks  compared  to  1  -Rank) . 

The  research  team  concluded  that  the  RIW 
experts  recommend  continued  use  by  the  Air 
Force  of  the  RIW  concept. 

Open-Ended  Questions 

The  fourth  section  of  the  interview  instrument,  Open 
Ended  Questions  (see  Appendix  B) ,  contained  two  questions 
which  afforded  the  RIW  experts  an  opportunity  to  comment  on 
their  RIW  programs,  and  to  comment  on  their  recommendation 
for  future  use  by  the  Air  Force  of  the  RIW  concept. 

To  ascertain  RIW  experts'  feelings  and  expressions 
of  intensity  about  the  RIW  concept,  the  research  team 
reviewed  and  interpreted  the  two  open-ended  questions. 


Appendix  D,  Tables  D-l  and  D-2,  contains  all  the  RIW 
experts'  responses  to  these  open-ended  questions.  With 
respect  to  the  first  open-ended  question,  the  following  is 
the  research  team's  review  and  interpretation  of  the  RIW 
experts'  comments  by  RIW  program: 

1.  AN/ARN  118  TACAN .  The  majority  of  comments 
indicated  possible  deficiencies  with  the  use  of  RIW  early 
in  program  development.  These  deficiencies  were  indicated 
by  comments  such  as  "the  AN/ARN  118  was  not  a  good  choice 
for  RIW,"  "reliability  was  already  built-in,"  and  "the  RIW 
drove  out  other  competitors." 

2.  C-141  AHRS.  Comments  on  this  program  indicated 
problems  in  the  management  of  the  RIW.  These  problems  were 
supported  by  comments  such  as  "accurate  MTBFs  could  not  be 
determined,"  and  "the  66-1  Maintenance  Data  Collection 
didn't  work." 

3.  OMEGA  Nav  Set.  Comments  such  as  "the  RIW 
enhanced  availability,"  OMEGA  was  a  good  choice  for  RIW," 
and  "the  RIW  provided  the  contractor  flexibility  to  make 
money  in  the  short  term  so  the  Air  Force  could  gain  bene¬ 
fits  in  the  long  term"  indicated  that  the  implementation  and 
use  of  RIW  on  this  program  was  positive. 

4.  Carousel  IV  INS.  Comments  on  this  program,  such 
as  "the  RIW  was  more  effective  getting  reliability  into  the 
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system,"  and  "the  contractor  had  high  confidence  in 
reliability  improvement"  illustrated  the  successful  applica¬ 
tion  of  RIW. 

5.  F-16  Electronic  Components.  Comments  on  this 
program  indicated  the  use  of  RIW  was  very  successful  with 
respect  to  system  cost  and  Interim  Contractor  Support  (ICS) 
cost  avoidance.  These  comments  were  "the  price  paid  for 
the  RIW  was  $50  -  $100  million  less  than  non-RIW,"  "cost- 
effectiveness  analysis  indicated  $100  million  savings  with 
the  RIW,"  and  "F-16  RIW  regarded  successful,  mostly  due  to 
ICS  cost  avoidance." 

6.  Air  Force  Standard  INU.  "Compared  to  the  non- 
RIW  INU  on  the  F-15,  the  Standard  INU  reliability  increased 
350  percent,"  and  "no-cost  ECPs  forced  reliability  growth  in 
the  first  two  years"  are  comments  which  illustrate  successful 
application  of  RIW  on  this  program. 

Regarding  the  second  open-ended  question,  the 
research  team's  review  and  interpretation  found  that  the 
majority  of  the  RIW  experts'  comments  recommend  continued 
use  of  the  RIW  concept  as  well  as  various  recommendations  for 
improving  the  concept.  Appendix  D,  Table  D-2,  contains 
these  RIW  experts'  recommendations. 

Summary 


The  Data  Analysis  Chapter  established  that  the  RIW 
experts  were  knowledgeable  and  experienced  (16.4  years  of) 


management  experience)  individuals,  each  with  an  average  of 
4.2  years  of  RIW  experience.  Nonparametric  statistical 
tests  were  used  to  analyze  the  two  research  questions  in 
Chapter  I.  Finally,  the  research  team  was  able  to  identify 


CHAPTER  V 


SUMMARY,  CONCLUSIONS  AND 
RECOMMENDATIONS 

Introduction 

The  purpose  of  this  chapter  is  to  present  the  results 
of  the  research  team’s  effort.  First,  the  research  team 
will  restate  the  research  objectives  and  questions  from 
Chapter  I  and  summarize  the  methodology  used  in  this 
research  effort.  Second,  the  research  team  will  present 
several  conclusions  drawn  from  the  research  effort.  Finally, 
recommendations  for  future  follow-on  research  efforts  and 
research  observations  relating  to  the  Air  Force  RIW  program 
will  be  identified. 


Research  Design  Summary 

The  two  research  objectives  and  questions  identified 
in  Chapter  I  established  the  basic  framework  for  this  research 
effort : 

Research  Objectives 

1.  To  determine  if  RIWs  applied  to  Air  Force  programs 
since  August  1974  were  effective. 

2.  To  recommend  the  future  use  by  the  Air  Force  of 


the  RIW  concept. 


Research  Questions 


1.  What  are  the  perceptions  of  active  duty  Air 
Force  officers  and  DOD  civilians  with  Air  Force  RIW  expertise 
regarding  the  effectiveness  of  RIWs  as  applied  to  Air  Force 
programs  since  August  1974? 

2 .  What  is  the  recommendation  for  the  future  use  by 
the  Air  Force  of  the  RIW  concept  as  perceived  by  Air  Force 
officers  and  DOD  civilians  with  Air  Force  RIW  expertise? 

Data  were  collected  by  means  of  personal/telephone 
interviews  with  the  identified  RIW  experts.  The  research 
team  analyzed  the  data  in  accordance  with  the  research 
methodology  described  in  Chapter  III.  The  main  points  of 
the  research  team's  methodology  included: 

1.  Selection  and  interview  of  RIW  experts. 

2.  Definition  and  recoding  of  RIW  experts'  ratings 
of  the  eleven  effectiveness  criteria. 

3.  Nonparametric  statistical  tests  used  to  answer 
Research  Question  One  which  consisted  of: 

a.  The  Wilcoxon  Matched-Pairs  Signed-Ranks  Test 
performed  on  RIW  experts'  median  ratings  of  the  effectiveness 
criteria. 

b.  The  Binomial  Test  performed  on  the  overall 
ratings  of  each  of  the  six  Air  Force  RIW  programs. 

4.  Nonparametric  statistical  tests  used  to  answer 
Research  Question  Two  which  consisted  of  the  Wilcoxon 


Matched-Pairs  Signed-Ranks  Test  performed  using  RIW  experts' 
recommendation  for  the  future  use  by  the  Air  Force  of  the  RIW 
concept. 

5.  Six  numerical  tables  that  presented  general/ 
demographic  information  on  the  RIW  experts. 

6.  Review  and  interpretation  of  RIW  experts' 
responses  to  the  two  open-ended  questions. 

Conclusions 

The  final  conclusions  of  this  research  effort  are 
presented  in  four  sections:  General /Demographic ,  Program 
Evaluation,  Future  Recommendation  and  Open-Ended  Questions. 
The  General /Demographic  section  contains  information  about 
RIW  experts'  career  areas  and  experience  levels.  The  Program 
Evaluation  and  Future  Recommendation  sections  address 
Research  Question  One  and  Two,  respectively.  Finally,  the 
Open-Ended  Questions  section  of  this  chapter  provides  addi¬ 
tional  feedback  from  the  RIW  experts  who  have  knowledge  and 
expertise  in  formulating,  implementing  and  monitoring 
Reliability  Improvement  Warranties. 

Before  accepting  the  conclusion  of  this  research 
effort,  ti a  reader  should  review  the  assumptions  and  scope 
and  limitations  presented  in  Chapter  III.  Further,  it  should 
be  noted  that  the  conclusions  apply  to  the  group  of  thirty- 
six  RIW  experts  who  participated  in  this  research  effort. 


71 


The  research  team  did  not  attempt  to  infer  or  imply  to  any 
overall  population. 


General /Demographic 

A  summary  of  the  research  team's  conclusions  relat¬ 
ing  to  the  demographic/general  profiles  of  RIW  experts  are 
listed  below: 

1.  The  research  team  concluded  that  the  majority 
(72.5  percent)  of  RIW  experts  come  from  the  general  career 
field  of  logistics  management.  For  this  research  effort, 
this  fact  shows  that  RIW  management  is  emphasized  in  the 
logistics  area. 

2.  The  combined  experience  level  for  all  RIW 
experts  in  their  respective  career  fields  is  16.4  years. 
However,  of  the  three  general  career  fields,  program  manage¬ 
ment  had  the  least  average  number  of  years  experience 

(8 . 7  years) . 

3 .  Seventy  percent  of  the  RIW  experts  with  program 
management  experience  were  active  duty  Air  Force  officers. 
One-hundred  percent  of  the  RIW  experts  with  contract  or 
logistics  management  experience  were  DOD  civilians. 

4.  The  combined  RIW  experience  level  for  all  RIW 
experts  in  their  respective  career  fields  is  4.2  years.  This 
is  twice  the  number  of  years  assumed  necessary  to  quality  as 


a  RIW  expert. 


5.  On  the  average,  RIW  experts  on  each  of  the  six 
Air  Force  RIW  programs  had  25.7  years  of  cumulative  RIW 
experience . 

Program  Evaluation 

A  summary  list  of  the  RIW  experts'  evaluations  of 
the  overall  effectiveness  of  the  Air  Force  RIW  program  is 
provided  below: 

1.  The  RIW  experts,  on  each  of  the  six  Air  Force 
RIW  programs,  perceived  that  the  use  of  the  RIW  on  their 
respective  program  was  effective.  This  conclusion  was 
statistically  supported  by  the  results  of  the  Wilcoxon 
Matched-Pairs  Signed-Ranks  Test  discussed  in  Chapter  IV. 

2.  Based  on  the  perceived  effectiveness  of  each 

of  the  six  Air  Force  RIW  programs,  the  RIW  experts  perceived 
that  the  overall  Air  Force  RIW  program  was  effective.  This 
conclusion  was  statistically  supported  by  the  results  of  the 
Binomial  Test  and  Wilcoxon  Matched-Pairs  Signed-Ranks  Test 
described  in  Chapter  IV. 

Future  Recommendation 

The  research  team  concludes  that  the  RIW  experts 
recommended  the  continued  use  by  the  Air  Force  of  the  RIW 
concept.  This  conclusion  is  statistically  supported  by  the 
results  of  the  Wilcoxon  Matched-Pairs  Signed-Ranks  Test 
described  in  Chapter  IV. 
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Open-Ended  Questions 

A  summary  list  of  the  research  team's  conclusions 
is  provided  below: 

1.  Although  the  RIW  experts  perceived  each  of  the 
six  Air  Force  RIW  programs  as  effective,  numerous  problems 
with  the  use  of  RIW  were  identified. 

2.  When  applied  effectively,  RIWs  can  be  used  as 
successful  contracting  incentives  both  in  the  acquisition 
and  support  of  Air  Force  systems. 

3.  For  the  RIW  concept  to  be  successfully  continued 
the  RIW  must  be  better  managed — the  right  system  needs  to 

be  selected  for  RIW  application,  accurate  and  sufficient 
data  on  the  system  needs  to  be  collected,  and  a  definitive 
contract  with  a  specific  RIW  clause  needs  to  be  applied  to 
the  system. 


Recommendations  for 
Future  Research 

The  research  team  suggests  the  following  recommenda¬ 
tions  for  future  research  in  descending  order  of  importance: 

1.  Replicate  this  project  using  industry's  RIW 
experts  on  each  of  the  six  Air  Force  RIW  programs. 

2.  For  RIW  management  purposes,  this  research 


effort  should  be  replicated  within  the  next  five  years  to 
evaluate  those  new  programs  with  RIWs. 


3.  A  parallel  research  effort  should  be  performed 
to  measure  the  effectiveness  of  the  Army  and  Navy's  RIW 
program  as  perceived  by  RIW  experts. 

4.  Perform  a  parallel  project  using  any  of  the  other 
Air  Force  Product  Performance  Agreements  (PPA)  to  measure 
the  effectiveness  of  that  agreement  as  perceived  by  that 
agreement's  experts. 

5.  Design  a  research  project  to  develop  a  standardized 
data  base  system  or  management  information  system  for  Air 
Force  RIW  management. 

6.  A  research  project  should  be  conducted  to  compare 
similar  systems  with  and  without  a  RIW. 

Research  Observations 

As  a  result  of  this  research  effort,  the  research 
team  has  shown  that  Reliability  Improvement  Warranties 
applied  to  Air  Force  programs  during  the  trial  period  were 
effective.  Additionally,  this  research  effort  indicated 
that  the  use  of  the  Air  Force  RIW  contracting  concept  should 
be  continued.  Of  practical  value,  this  research  effort 
provided  Air  Force  management  the  following: 

1.  Real-time  feedback  from  RIW  experts  regarding 
RIW  program  effectiveness. 

2.  An  assessment  of  the  Air  Force  RIW  trial  period. 

3.  Justification  for  continued  use  by  the  Air  Force 
of  the  RIW  concept. 
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DEPARTMENT  OF  THE  AIR  FORCE 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  IATC1 
WRIGHT  PATTERSON  AIR  FORCE  BASE.  OH  «S433 


5  May  1933 

**£.’=?  ISO  ( LS SR  53-33/Capts  0.  Parkinson  and  A.  Schoolcraf t/AV  785-6569 

sutler  Thesis  Research  -  Effectiveness  of  Reliability  Improvement  Warranties  (RIW) 


1.  The  purpose  of  this  letter  is  to  thank  you  for  your  willingness  to  par¬ 
ticipate  in  this  research,  and  to  provide  you  preliminary  information  prior 
to  your  interview  with  the  research. team. 

2.  You  have  been-  identified  by  the  research  team,  and  by  your  peers,  as  hav¬ 
ing  knowledge  of  RI'Ws  as  applied  to  specific  Air  Force  programs.  Your 
knowledge  and  experience  provided  during  the  interview  are  very  important  to 
this  research  project.  Your  responses  will  be  kept  anonymous,  both  within 
the  group  of  interviewees  and  in  the  report. 

3.  Attachment  1  provides  additional  information  about  this  research  project, 
your  participation  in  the  research,  and  your  preparation  for  the  interview. 
This  information  will  give  you  time  to  prepare  for  the  interview  as  well  as 
reduce  the  length  of  the  interview. 

4.  Captains  Parkinson  and  Schoolcraft  look  forward  to  meeting  with  you. 

They  will  be  contacting  you  within  a  week  to  schedule  a  mutually  convenient 
time  for  the  interview. 

THE000RE  J.  NC'/AK,  Jr.,  it  Col,  USAF 
Head,  Dept  of  Cost  Analysis  and  Pricing 
School  of  Systems  and  Logistics 


1  Atch 

Interviewee  Information 
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INTERVIEWEE  INFORMATION 


The  position  you  hold  (or  held)  within  your  program 
identifies  you  to  be  among  the  most  knowledgeable  individuals 
with  Air  Force  RIW  experience.  We  believe  that  your  responses 
will  help  us  analyze  the  impact  RIWs  have  had  during  the 
Air  Force  RIW  trial  period. 

The  purpose  of  this  research  is  to  evaluate  the  effec¬ 
tiveness  of  RIWs  applied  to  Air  Force  programs  since  1974, 
and  to  make  recommendations  as  to  the  future  use  of  the  Air 
Force  RIW  concept. 

During  the  interview,  you  will  be  asked  to  rate  your 
program  (see  Atch  1-2)  against  each  of  the  effectiveness 
criteria  defined  in  Atch  1-3. 

By  providing  this  information  prior  to  the  interview,  we 
feel  you  can  briefly  review  these  criteria  and  your  program, 
and  thus  be  better  prepared  to  evaluate  your  program's  RIW 
during  the  actual  interview. 

We  expect  the  interview  to  last  no  more  than  15-20  minutes. 
After  completion  of  the  structured  interview,  feel  free  to 
provide  open  comments  about  your  program.  If  you  have  any 
questions  prior  to  the  interview,  please  call  either  of  the 
researchers . 


Atch  1-1 
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PROGRAM  RATING 


With  respect  to  your  RIW  program,  please  use  the 
following  scale  and  question  to  rate  each  of  the  effec¬ 
tiveness  criteria. 

1  2  3  4  5  6  7 

I - 1 - i - 1 - 1 - 1 - * 

LEAST  LESS  SLIGHTLY  SAME  SLIGHTLY  MORE  MOST 

LESS  AS  MORE 

Questions  Based  on  your  experience  and  expertise,  how 

would  you  rate  your  program  with  RIW  compared 
to  similar  programs  without  RIW? 


Reliability 
Maintainability 
Acquisition  cost 
Equipment  modification 
Data  collection 
Contractor  competition 
Use  rate 
Supply  system 
Maintenance  skill 
Number  of  spares 
RIW  contract  clause 


Atch  1-2 


80 


EFFECTIVENESS  CRITERIA  DEFINED 


Reliability  -  the  probability  that  an  item  will  perform  its 
intended  function  for  a  specified  interval  under  stated 
conditions . 

Maintainabil ity  -  the  probability  that  an  item  will  be  restored 
to  a  specified  condition  within  a  given  period  of  time 
when  proper  maintenance  is  performed. 

Acquisition  cost  -  the  total  expenditures  incurred  for  the 
purchase  or  production  of  an  item. 

Equipment  modification  (ECN)  -  alterations  in  the  physical  or 
functional  characteristics  of  an  item  after  initial 
establishment  of  these  characteristics. 

Data  collection  -  collection  of  required  data  (R&D,  production, 
training,  modification,  overhaul,  O&M,  etc.)  during  item 
procurement  and  support  by  all  levels  of  management. 

Contractor  competition  -  competition  between  responsible, 
responsive  contractors  during  all  phases  of  item 
acquisition  and  support. 

Use  rate  -  the  quantity  of  usage  by  type  that  occurs  in  a 
specific  time  period. 

Supply  system  -  organizations,  •  rices,  facilities,  methods, 

and  techniques  necessary  to  provide  an  item  to  authorized 
users . 

Maintenance  skill  -  skill  required  at  all  maintenance  levels 
to  provide  proper  and  adequate  item  repair. 

Number  of  spares  -  articles  identical  to  or  interchangeable 
with  the  end  item  which  are  procured  over  and  above  the 
quantity  needed  for  initial  installation. 

RIW  Contract  clause  -  content  necessary  to  insure  that  required 
acquisition  and  support  goals  and  objectives  are 
established  for  procurement  of  an  item. 

Note:  The  above  effectiveness  criteria  are  the  result  of  a 

1980  AFIT  research  study  (LSSR  63-80) .  Also,  the 
listed  order  has  no  significance. 
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CAREER  FIELD  OF  MOST  EXPERIENCE  (AFSC)  _ 

YEARS  IN  CAREER  FIELD  _ 

YEARS  ON  RIW  PROGRAM  _ 

OTHER  RIW  EXPERIENCE  (IDENTIFY  PROGRAM)  _ 

PARTICIPATION  IN  OTHER  RIW  RESEARCH  PROJECTS  (IDENTIFY 
PROJECTS)  _ 


PROGRAM  EVALUATION 


With  respect  to  your  RIW  program,  please  use  the  following 
scale  and  question  to  rate  each  of  the  effectiveness  criteria. 


1  2  3  4  5  6  7 

, - 1 - 1 - 1 - 1 - 1 - 1 

LEAST  LESS  SLIGHTLY  SAME  SLIGHTLY  MORE  MOST 

LESS  AS  MORE 

Questions  Based  on  your  experience  and  expertise,  how  would 
you  rate  your  program  with  RIW  compared  to 
similar  programs  without  RIW? 

_  Reliability 

_  Maintainability 

_  Acquisition  cost 

_  Equipment  modification 

_  Data  collection 

_  Contractor  competition 

_  Use  rate 

_  Supply  system 

_  Maintenance  skill 

_  Number  of  spares 

_  RIW  contract  clause 
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FUTURE  RECOMMENDATION 


With  respect  to  the  Air  Force  RIW  program,  please  use 
the  following  scale  and  question  to  recommend  the  future 
use  by  the  Air  Force  of  the  RIW  concept. 

12  3 

I - - - 1 - 1 

Discontinue  Undecided  Continue 

Question*  Based  on  your  experience  and  expertise,  what  would 
be  your  recommendation  on  the  future  use  by  the 
Air  Force  of  the  RIW  concept? 


OPEN-ENDED  QUESTIONS 


1.  Do  you  have  any  comments  about  your  .RIW  program  (good/bad) 
that  you  feel  this  research  effort  did  not  address? 


2.  Based  on  your  recommendation  on  the  future  use  of  RIW's 
in  the  Air  Force,  what  comments  do  you  have  to  support 
this  decision? 


TABLE  C-l.  RESULTS!  RESEARCH  HYPOTHESIS  ONE 


HILCOXON  MATCHED-PAIRS  SICNED-RANKS  TEST 


CRITERIA  OF  THE  M-ARN  118  TACAN  RIU  PROGRAM ' 
WITH  NEUTRAL 


17  -RANKS  33  ♦RANKS 
CASES  TIES  MEAN  MEAN 

44  14  24.32  24.11 


2 

-2.142 


TABLE  C-2.  RESULTS:  RESEARCH  HYPOTHESIS  TliO 


-  HILCOXON  HATCHED-PAIRS  SICNED-RANKS  TEST 

CRITERIA  OF  THE  C-141  AHRS  RIU  PROGRAM 
WITH  NEUTRAL 


13  -RANKS  33  ♦RANKS 
CASES  TIES  MEM  MEM 

SS  9  18.19  25.59 


Z 

-3.321 


TABLE  C-3.  RESULTS!  RESEARCH  HYPOTHESIS  THREE 


-  HILCOXON  MATCHED-PAIRS  SICNED-RANKS  TEST 

CRITERIA  OF  THE  OMEGA  NAV  SET  RIH  PROGRAM 
WITH  NEUTRAL 


2f  -RANKS  39  ♦RANKS 
CASES  TIES  MEM  MEM 

77  18  30.15  29.92 


Z 

-2.129 


2-TAILED  P 
.031 


2-TAILED  P 

.001 


2-TAILED  P 
.033 
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TABLE  C-4.  RESULTS:  RESEARCH  HYPOTHESIS  FOUR 


-  -  -  HILCOXOH  HATCHED-PAIRS  SICNED-RANKS  TEST 


CRITERIA 

HITH  NEUTRAL 

OF  THE  CAROUSEL  IV  INS  RIH  PROCRAH 

CASES 

55 

TIES 

5 

17  -RANKS 
HEAN 
22.16 

33  4RANKS 

HEAN 

27.27 

Z 

-2.534 

2-TAILED  P 
.111 

TABLE  C-5.  RESULTS:  RESEARCH  HTPOTHESIS  FIVE 


-  HILCOXOH  HATCHED-PAIRS  SICNED-RANKS  TEST 


CRITERIA  OF  THE  F-16  ELECTRONIC  COKP  RIH  PROCRAH 
WITH  NEUTRAL 


CASES 

77 


22  -RANKS  43  ♦RANKS 
TIES  HEAN  MEAN 

12  31.73  34.16 


Z  2-TAILED  P 

-2.591  .111 


TABLE  C-6.  RESULTS:  RESEARCH  HTPOTHESIS  SIX 

-  HILCOXON  HATCHED-PAIRS  SICNED-RANKS  TEST 

CRITERIA  OF  THE  AF  STANDARD  INU  RIH  PROCRAH 
WITH  NEUTRAL 


CASES 


TIES 


15  -RANKS  41  +RANKS 
KEAN  HEAN 


Z 


2-TAILED  P 


TAHi  C-7.  RESULTS:  RESEARCH  HYPOTHESIS  SEVEN  (TEST  1) 


•  -  BINOMIAL  TEST  -  PROBABILITIES  FOR  N  *  4 


NUMBER  OF  EFFECTIVE 

RIH  PROGRAMS 
(X) 

ONE-TAILED 

PROBABILITY 

P(X) 

CUMULATIVE 

PROBABILITY 

« 

.114 

.114 

l 

.194 

.111 

2 

.234 

.344 

3 

.313 

.457 

4 

.234 

.891 

5 

.194 

.985 

4 

.014 

1.001 

TABU  C-8.  RESULTS:  RESEARCH  HYPOTHESIS  SEVEN  (TEST  21 


-  HILCOXON  HATCHED-PAIRS  SICNED-RANKS  TEST 

PROCRAH  OF  OVERALL  RIM  EFFECTIVENESS 
WITH  NEUTRAL 


CASES 

TIES 

114  -RAWS 
MEAN 

222  +RANKS 

MEAN 

Z 

2-TAILED  P 

394 

7« 

149.42 

17f.ll 

-4.524 

.m 

TABU  C-9.  RESULTS:  RESEARCH  HYPOTHESIS  EIGHT 


HILCOXON  HATCHED-PAIRS  SICNED-RANKS  TEST 


RECQHHEN 

WITH  NEUTRAL 

FOR  THE  FUTURE  OF  THE  RIH  CONCEPT 

CASES 

34 

TIES 

5 

1  -RAWS 
MEAN 
14.11 

30  ♦ RANKS 

MEAN 

14. N 

Z 

-4.544 

2-TAILED  P 

.Ml 

APPENDIX  D 

COMMENTS  ON  OPEN-ENDED  QUESTIONS 


Table  D-l.  Open-Ended  Comments  by  Program 


AN/ARN  118  TACAN 

1.  The  AN/ARN  118  was  not  a  good  choice  for  RIW. 

2.  System  reliability  was  already  built-in  prior  to 
contract  award. 

3.  There  was  only  one  ECP  submitted  early  in  the 
program. 

4.  The  use  of  RIW  drove  out  other  competitors. 

5.  Even  with  higher  reliability,  Warner-Robins  ALC 
spared  at  the  normal  20-22%  level. 

6.  The  use  of  RIW  drove  the  contractor  to  design  in 
more  reliability. 


C-141  AHRS 

1.  The  Air  Force  could  not  accurately  determine  the 
MTBF  of  the  C-141  AHRS. 

2.  Failure  data  was  collected  using  the  66-1  Maintenance 
Data  Collection  System.  This  system  did  not  work  with 
AHRS  RIW. 

3.  Together  with  3-level  maintenance  and  the  RIW,  we 
could  not  key  on  specific  end  item  failures. 

4.  Many  intangible  benefits  resulted  from  using  the 
RIW.  One  example  was  that  the  contractor  identified 
other  aircraft  problems  that  impacted  on  his  RIW 
system. 


OMEGA  NAV  SET 

1.  OMEGA  was  a  good  choice  for  RIW. 

2.  Dynell  was  a  good,  responsive  company  to  work  with. 

3.  Due  to  complex  software  and  antennae,  there  were 
more  ECPs  than  usual. 

4.  Using  RIW  enhanced  the  availability  of  OMEGA. 
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5.  OMEGA  RIW  was  probably  the  best  in  the  Air  Force  as 
indicated  by  the  positive  attitude  of  the  contractor, 
the  good  data  collection  by  MAC,  and  the  reliability 
of  that  type  of  equipment. 

6.  Use  of  RIW  gives  the  contractor  the  flexibility  to 
make  money  in  the  short  term  so  that  the  Air  Force 
can  benefit  in  the  long  term. 

7.  There  was  too  much  administration  of  the  RIW  for 
government  and  contractor  personnel. 

8.  There  were  too  many  ECPs  submitted  for  which  the 
government  had  to  pay. 

9.  The  RIW  clause  for  contingency  spares  was  too  lo^se. 


CAROUSEL  IV  INS 

1.  On  the  KC-135,  the  Air  Force  continually  changec 
integration  requirements  which  caused  changes  ii 
software.  This  in  turn  affected  the  RIW. 

2.  The  RIW  was  more  effective  getting  reliability  into 
the  system. 

3.  Delco  had  high  confidence  in  reliability  improvement 
as  evidenced  by  good  internal  procedures  to  insure 
reliability  and  prior  commercial  airline  experience. 


F-16  ELECTRONIC  COMPONENTS 

1.  When  the  F-16  RIW  was  being  considered  (1976), 
there  was  little  or  no  other  RIW  experience  for 
comparison. 

2.  Because  RIW  options  were  firm  fixed  price  options, 
obtained  in  a  competitive  source  selection,  these 
prices  made  subsequent  decisions  to  exercise  contract 
options  much  simpler. 

3.  Since  F-16  RIW  components  were  complex  avionics,  the 
Air  Force  could  not  achieve  organic  support  for  several 
years.  Thus,  the  decision  was  to  either  buy  sole 
source  interim  contractor  support  or  buy  the  RIW  in 

a  competitive  environment.  Once  the  ICS  prices  were 
known,  the  selection  of  RIW  was  evident. 


4.  A  RIW  cost  effectiveness  analysis  indicated  the  Air 
Force  oould  save  $100  million  on  the  nine  LRUs  selected 
for  RIW  application. 

5.  Unconfirmed  reports  indicated  the  RIW  yielded  very 
little  reliability  improvement. 

6.  The  F-I6  RIW  has  been  regarded  as  successful, 
primarily  due  to  ICS  cost  avoidance. 

7.  The  incentive  to  have  the  contractor  improve  reliability 
was  not  as  good  as  planned. 

8.  The  price  paid  for  the  RIW  systems  was  $50-100 
million  less  than  non-RIW  systems. 


AIR  FORCE  STANDARD  INU 

1.  Good  data  collection  is  the  key  to  successful  RIW 
management . 

2.  RIW  requires  a  full-time  manager. 

3.  With  RIW,  the  program  office  gets  good  visibility 
into  the  program. 

4.  Under  this  RIW,  all  ECPs  were  no  cost  to  the 
government . 

5.  Compared  to  the  non-RIW  INU  on  the  F-15,  the 
standard  INU  reliability  increased  350%. 

6.  Reliability  growth  was  forced  into  the  system  in  the 
first  two  years  through  29  no-cost  ECPs. 

7.  A  key  to  a  successful  RIW  is  a  good  technical 
specification. 

8.  The  standard  INU  is  46%  cheaper  than  the  original 
F -16  INU. 
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Table  D-2.  Open-Ended  Comments  on  Future  Recommendations 


1.  RIWs  should  be  considered  for  new  technology  items. 

2.  RIWs  should  be  considered  on  smaller  programs. 

3.  RIWs  should  be  considered  on  simpler  systems. 

4.  RIWs  should  be  considered  if  the  right  equipment  is 

selected  for  RIW  application.  That  is  the  equipment 
should  have  initial  low  reliability  with  the  potential 
for  improved  reliability. 

5.  RIWs  should  be  considered  for  avionics  programs. 

6.  RIWs  should  be  priced  in  a  competitive  environment. 

7.  RIWs  should  be  applied  to  programs  with  long 

production  runs. 

8.  RIWs  should  be  applied  to  black  box  (LRU)  items  only. 

9.  RIW  is  a  good  concept  and  the  Air  Force  should  push 
for  continued  use. 

10.  We  should  put  cost  to  benefit  gained. 

11.  We  should  contract  for  two  units — one  with  and  one 
without  RIW. 

12.  The  Air  Force  should  continue  using  RIWs  because 
RIWs  have  made  more  money  for  the  Air  Force  than  any 
other  procurement  program. 

13.  Program  offices  should  get  item  managers  and  system 
managers  involved  early  in  formulating  RIWs. 

14.  The  Air  Force  should  demand  what  it  wants  from  the 
RIW.  We  cannot  continue  to  assume  the  contractor 
will  give  us  improved  reliability  if  we  do  not  ask 
for  it. 

15.  The  Air  Force  should  specifically  define  what  a 
failure  is  in  the  contract  provisions. 

16.  We  should  continue  the  use  of  RIWs  if  the  program 
has  a  definitive  contract  with  a  strong  RIW  clause. 


17.  The  Air  Force  should  develop  a  data  collection  system 
designed  specifically  to  track  RIW  data. 


We  should  apply  RIWs  more  extensively. 

The  Air  Force  should  insure  that  the  contractor  does 
not  suboptimize  his  internal .  operations.  Tha.t  is, 
we  should  not  force  him  to  place  priority  on  RIW 
items  at  the  expense  of  other  non-RIW  items. 

To  make  RIWs  work  eff  rtively,  the  Air  Force  should 
insure  that  adequate  time,  money,  and  personnel  are 
allocated  to  monitor  the  RIW. 
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